What's needed?

Coordinator
Jan 27, 2009 at 5:24 PM
We're looking to bring SolrSharp up to feature-speed with Solr 1.3 in the near term.  In process of doing that, we'll likely refactor some of the existing code.

As will naturally happen, we'll soon discover better ways of making the code work as well as seeing opportunities for new features within SolrSharp.  To guide that conversation, it's always helpful to know what others would like to see in the library.

So, what is the library missing?  If you feel there's a feature that's needed, please explain it with some justification.  Even better, submit a patch.  ;-)
Jan 27, 2009 at 8:10 PM
I've only started playing with SolrSharp. (Literally, just last Thursday.) So take these comments with some salt:
  • About the only major thing I had to update was the various URLs for Solr 1.3. I guess this could be moved to configuration.
  • Doing a lot of work in constructors has the unfortunate side effect that any exception results in a TypeInitializerException that wraps the real exception and makes debugging a little more difficult than it needs to be. But I would reckon that it's too late to change the general design patterns used. And it's not that bad. I'm just complaining.
  • The IndexFieldAttribute reflection magic seems to not like it when a multivalued field is sometimes returned with one value (tries to set a string property) versus many values (tries to set a string[] property). But I've found I've been doing a lot of my initialization in the constructor anyway and using that attribute as just a shortcut for the really simple properties.
That's it so far! Thanks for this great library.
Jan 28, 2009 at 3:54 PM
Hmm, also, the library seems to be missing support for facet queries. AddSearchParameter() doesn't work because if you call it multiple times, it keeps overwriting the value.

Should there be a FacetQuery object?
Coordinator
Jan 28, 2009 at 4:07 PM
Great points for sure, thanks for the comments.  When I started the project, the goal was to provide access to every Solr feature through the client lib, and that goal remains intact.
Feb 2, 2009 at 9:48 PM
I would second the proposition of better error message bubbling up to the consumers of the client library. Right now the only I see is that generic server error response, but no clue what happened there in detail. I will take a look into that tonight and see if I can analyze what code changes  can be done.

As for npiaseck and facet support, QueryBuilder in solrsharp has AddFacet method which accepts an instance of a class that inherits from Facet class. You can specify which field this facet will be for. Have you tried that?
Feb 2, 2009 at 9:53 PM
Also, not sure if I should have proposed this as a patch or what, but I would like to see an alternative to configuring SolrSharp besides app.config file (talking about 'solr' configuration section).

I modified the SolrSearchers class to not fail with the exception when 'solr' section is not found (it tries to read it, but moves on if it is not there) and exposed AddSearcher method. That way I can have external component configuring the solrsearch. So basically there is a choice, use config or do it manually if you need it.
Feb 2, 2009 at 9:56 PM
Laimis,

I indeed saw the support for regular facets via the AddFacet and Facet class, but I didn't see any native support for facet queries, where you do things like "actualPrice:[0 to 9.99]", "actualPrice:[10 TO 19.99]", "actualPrice:[20 to *]" and get those facet counts returned in the facet_queries bit of the XML response. I added support for this in my implementation by just fudging a few methods to be protected virtual so I could override them and grab that part from the response.

If I overlooked a FacetQuery (not facet) object entirely, I'll sure feel silly!
Feb 2, 2009 at 10:10 PM
> I indeed saw the support for regular facets via the AddFacet and Facet
> class, but I didn't see any native support for facet queries

My bad, didn't read your post carefully enough. Yes, agreed, having
facet QUERIES would be nice.
Feb 5, 2009 at 11:33 PM
Another thing I thought that SolrSharp could do better is logging. I was thinking something similar along the lines of NHibernate and how it uses log4net to output information to the log files. For instance turning on info logging SolrSharp could output the query it is just about to pass to SOLR instance, etc.
Coordinator
Feb 15, 2009 at 7:21 PM
an alternative to configuring SolrSharp besides app.config file (talking about 'solr' configuration section).

If I follow you correctly, it sounds as if you're asking for silent failure (or simply no initialization.) Similarly to log4net, if I'm making the correct interpretation.  There are other considerations to make in that context, i.e. when you want to enforce configuration or else the application fails.

> better error message bubbling up to the consumers

I've been exploring this a bit, and I'm not prepared to make any suggestions, but there are a few key things that should be addressed:
- Published events that implementations can observe/subscribe to that allow for introspection into solr request/response operations
- Better server response intepretation.

> native support for facet queries

Yep.

> logging

Kind of along the lines of published events. Adding support for log4net would be helpful, though I know a lot of developers who don't always use log4net. Don't want to get to abstract-y, but it would be good to support more logging choices aside from log4net.
Feb 23, 2009 at 12:50 PM
How far along is the code under development? 

We (a company) are looking to switch a product currently using a commercial search engine to use Solr instead, but but will need to query from .NET, and will need to do faceted queries.  

Will we need to chip in some code to get drill-down facets in our results?

Without new features like facets, how much do we have to do to get the existing SolrSharp features to work with Solr 1.3 and 1.4dev? 

Jeffro - are you creating a branch for Solr 1.3 support, since the path changes will probably slightly-break SolrSharp for Solr 1.2?

Feb 23, 2009 at 1:09 PM
Edited Feb 23, 2009 at 1:13 PM
I added support for FacetQueries in a mostly non-obtrusive way by extending the library a little bit. The main things were that I...

...created an abstract FacetQuery class...

{
    using System;
    using System.Collections.Generic;
    using System.Web;

    /// <summary />
    /// A base class for facet queries.
    /// 
    public abstract class FacetQuery
    {
        /// <summary />
        /// The parameter key for facet queries.
        /// 
        private static readonly string SolrFacetQuery = "facet.query";

        /// <summary />
        /// The list of facet queries.
        /// 
        private List<string /> queries;

        /// <summary />
        /// Initializes a new instance of the FacetQuery class.
        /// 
        public FacetQuery()
        {
            this.queries = new List<string />();
        }

        /// <summary />
        /// Gets the name of the field that is being faceted.
        /// 
        public abstract string Field { get; }

        /// <summary />
        /// Gets a list of the queries that have been added to the FacetQuery
        /// instance.
        /// 
        public string[] Queries
        {
            get { return this.queries.ToArray(); }
        }

        /// <summary />
        /// Gets syntactically structured parameters for use within a URL as part
        /// of an HTTP search request.
        /// 
        public string UrlParameters
        {
            get
            {
                string[] parameters;

                parameters = new string[this.queries.Count];
                for (int i = 0; i < parameters.Length; ++i)
                {
                    parameters[i] = SolrFacetQuery + "=" + HttpUtility.UrlEncode(this.queries[i]);
                }

                return string.Join("&", parameters);
            }
        }

        /// <summary />
        /// Adds the facet query to the list of queries.
        /// 
        /// <param name="query" />the query to add to the facet query
        protected void AddFacetQuery(string query)
        {
            if (query == null)
            {
                throw new ArgumentNullException("query");
            }

            this.queries.Add(query);
        }
    }
}
...created an IFacetResults<T> interface to bridge between SolrSharp's native FacetResult class and my custom FacetQueryResults classes...

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;

    /// <summary />
    /// An interface that indicates that the item contains results about a facet
    /// or a facet query.
    /// 
    /// <typeparam name="T" />the type of the facet.
    public interface IFacetResults<t />
    {
        /// <summary />
        /// Gets a list of the facet results and the quantities available.
        /// 
        List<keyvaluepair<t, />> Facets { get; }
    }
...and then about the only direct change to SolrSharp itself is that I fudged the end of SearchResults<T>'s ExecuteSearch() method to include a call to a new abstract method called InspectResponse() which receives the entire XML response. This allows me to inspect the entire response in my derived class and add support for facet queries and spelling suggestions without making huge changes to SolrSharp itself, which would've only caused my heartburn if I ever tried to upgrade it down the line. Here's what that method looks like in my derived ProductSearchResults class, for example:

    using System;
    using System.Collections.Generic;
    using System.Xml;
    using org.apache.solr.SolrSharp.Query;
    using org.apache.solr.SolrSharp.Results;

    /// <summary />
    /// Search results obtained from a search by a ProductQueryBuilder.
    /// 
    public class ProductSearchResults : SearchResults<productsearchrecord />
    {
        /// <summary />
        /// A listing of facet results.
        /// 
        private List<ifacetresults<string />> facetResults = new List<ifacetresults<string />>();

        /// <summary />
        /// A list of spelling suggestions.
        /// 
        private List<spellingsuggestion /> spellingSuggestions = new List<spellingsuggestion />();

        /// <summary />
        /// Initializes a new instance of the ProductSearchResults class.
        /// 
        /// <param name="builder" />the query builder that created the instance
        public ProductSearchResults(QueryBuilder builder)
            : base(builder)
        {
            if (!(builder is ProductQueryBuilder))
            {
                throw new ArgumentOutOfRangeException("builder", "Must be of type ProductQueryBuilder.");
            }
        }

        /// <summary />
        /// Gets a list of the facet results.
        /// 
        public IFacetResults<string />[] FacetResults
        {
            get { return this.facetResults.ToArray(); }
        }

        /// <summary />
        /// Gets a list of spelling suggestions. Never null, but can be empty.
        /// 
        public SpellingSuggestion[] SpellingSuggestions
        {
            get { return this.spellingSuggestions.ToArray(); }
        }

        /// <summary />
        /// Initializes the facet results based on the given XML node.
        /// 
        /// <param name="xn" />the XML node to evaluate
        public override void InitFacetResults(XmlNode xn)
        {
            if (xn == null)
            {
                throw new ArgumentNullException("xn");
            }

            this.AddFacetResults(new BrandFacetResults(xn));
            this.AddFacetResults(new StyleFacetResults(xn));
            this.AddFacetResults(new ColorFacetResults(xn));
            this.AddFacetResults(new SizeInStockFacetResults(xn));
            this.AddFacetResults(new FabricFacetResults(xn));
            this.AddFacetResults(new LineFacetResults(xn));
            this.AddFacetResults(new ProductStockStatusFacetResults(xn));
            this.AddFacetResults(new ProductActiveStatusFacetResults(xn));
        }

        /// <summary />
        /// Creates a product search record.
        /// 
        /// <param name="xn" />the XML node to evaluate
        /// <returns />the correct search record object
        protected override ProductSearchRecord InitSearchRecord(XmlNode xn)
        {
            return new ProductSearchRecord(xn);
        }

        /// <summary />
        /// Nick's custom hook to allow me to examine suggested spelling,
        /// facet queries, etc.
        /// 
        /// <param name="response" />the XML document that was sent by the Solr
        /// instance
        protected override void InspectResponse(XmlDocument response)
        {
            XmlNode xn;

            base.InspectResponse(response);

            // Add Facet Query Results for Price
            xn = response.SelectSingleNode(@"response/lst[@name='facet_counts']/lst[@name='facet_queries']");
            this.AddFacetResults(new ActualPriceFacetQueryResults(xn));

            // Add spelling suggestions here
            xn = response.SelectSingleNode(@"response/lst[@name='spellcheck']/lst[@name='suggestions']");
            if (xn != null)
            {
                foreach (XmlNode spellNode in xn.ChildNodes)
                {
                    string misspelled;
                    SpellingSuggestion spellingSuggestion;
                    XmlNodeList suggestions;

                    misspelled = spellNode.Attributes["name"].Value;
                    suggestions = spellNode.SelectNodes(@"arr[@name='suggestion']/str");

                    spellingSuggestion = new SpellingSuggestion(misspelled);
                    foreach (XmlNode suggestion in suggestions)
                    {
                        spellingSuggestion.AddSuggestion(suggestion.FirstChild.Value);
                    }

                    this.spellingSuggestions.Add(spellingSuggestion);
                }
            }
        }

        /// <summary />
        /// Adds a facet result to the internal list of facet results if the
        /// number of facets returned is greater than zero.
        /// 
        /// <param name="facetResults" />the facet results
        private void AddFacetResults(IFacetResults<string /> facetResults)
        {
            if (facetResults.Facets.Count > 1)
            {
                this.facetResults.Add(facetResults);
            }
        }
    }

Hope that helps! No idea how the guys are actually going to implement these things, but this enabled me to get things up and running on Solr 1.3 last month while still being able to use most of the current library. Good luck!

(And a completely unrelated rant...the text editor on Codeplex is simply awful!)

Coordinator
Feb 24, 2009 at 5:36 AM
@twozen: are you creating a branch for Solr 1.3 support?

Yes, we will create a branch for 1.3.

@npiaseck: Thanks!  

Feb 27, 2009 at 5:58 PM
your document and field objects need a 'boost' property.
Feb 27, 2009 at 6:13 PM
Note to npiaseck (just FYI)

You can add facet queries to the request handlers in SolrConfig.xml:

        <lst name="invariants">
            <str name="facet.field">cat</str>
            <str name="facet.field">manu_exact</str>
            <str name="facet.query">price:[* TO 500]</str>
            <str name="facet.query">price:[500 TO *]</str>
        </lst>
    </requestHandler>

Feb 27, 2009 at 6:23 PM
> your document and field objects need a 'boost' property.

+1

I needed to change classes a bit to get document level boosting. It
was rather hackish though due to the way the UpdateIndexDocument class
is laid out and serialized.

L

>
> Read the full discussion online.
>
> To add a post to this discussion, reply to this email
> (solrsharp@discussions.codeplex.com)
>
> To start a new discussion for this project, email
> solrsharp@discussions.codeplex.com
>
> You are receiving this email because you subscribed to this discussion on
> CodePlex. You can unsubscribe or change your settings on codePlex.com.
>
> Please note: Images and attachments will be removed from emails. Any posts
> to this discussion will also be available online at codeplex.com
Coordinator
Mar 2, 2009 at 5:15 PM
> your document and field objects need a 'boost' property.

+1

I needed to change classes a bit to get document level boosting. It
was rather hackish though due to the way the UpdateIndexDocument class
is laid out and serialized.

Yep, agreed. Thanks for the comments, these are a big help.
Mar 2, 2009 at 9:15 PM
Hey Just another FYI, here is how I improved solrsharp error handling.  this will bubble solr errors out to the user by fixing a masking null reference error occuring in the catch clause when there is no response.

Here is the stack trace and the code i wrote to handle this error:

An exception of type 'System.NullReferenceException' occurred and was caught.
02/10/2009 13:59:26
Type : System.NullReferenceException, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
Message : Object reference not set to an instance of an object.
Source : SolrSharp
Help link :
Data : System.Collections.ListDictionaryInternal
TargetSite : System.Xml.XmlDocument GetXmlDocumentFromPost(System.String, System.String)
Stack Trace :    at org.apache.solr.SolrSharp.Configuration.SolrSearcher.GetXmlDocumentFromPost(String url, String queryparameters) in C:\TFSSC\3rd Party\SOLRSharp\src\Configuration\SolrSearcher.cs:line 341
   at org.apache.solr.SolrSharp.Results.SearchResults`1.ExecuteSearch(QueryBuilder queryBuilder) in C:\TFSSC\3rd Party\SOLRSharp\src\Results\SearchResults.cs:line 91
   at Buy.WebServices.SS.SearchServer.SearchEngine.Solr.BuySearchResults..ctor(List`1 contentAttributeIds, BuyQueryBuilder queryBuilder) in C:\TFSSC\Search Systems - SS\Solr Search\DEV\FB001\Search Engine\SearchServer.SearchEngine.Solr\BuySearchResults.cs:line 32
   at Buy.WebServices.SS.SearchServer.SearchEngine.Solr.SolrSearchEngine.InnerSearch(String phrase, SearchFilters filters, Sort sort, SortOrder sortOrder, Int32 pageNumber, Int32 pageSize, List`1 desiredPriceRanges, List`1 requestedContentAttributeFacets, SearchMetadata& metadata) in C:\TFSSC\Search Systems - SS\Solr Search\DEV\FB001\Search Engine\SearchServer.SearchEngine.Solr\SolrSearchEngine.cs:line 88
   at Buy.WebServices.SS.SearchServer.SearchEngine.BasicSearchEngine.FacetDBConnectedSearchEngine.Search(String phrase, SearchFilters filters, Sort sort, SortOrder sortOrder, Int32 pageNumber, Int32 pageSize, List`1 desiredPriceRanges, SearchMetadata& metadata) in C:\TFSSC\Search Systems - SS\Search Server\DEV\FB001\Search Server\SearchServer.SearchEngine\BasicSearchEngine\FacetDBConnectedSearchEngine.cs:line 208
   at Buy.WebServices.SS.SearchServer.SearchService.Search(String phrase, SearchFilters filters, Sort sort, SortOrder sortOrder, Int32 pageNumber, Int32 pageSize, List`1 desiredPriceRanges, SearchMetadata& metadata) in c:\TFSSC\Search Systems - SS\Search Server\DEV\FB001\Search Server\SearchServer\App_Code\SearchService.cs:line 214


Added Following Code to SolrSearcher.cs:line 341:

                WebResponse webResponse = e.Response;
                if (webResponse == null)
                    errsr = "No Response " + e.Message + Environment.NewLine + e.InnerException.Message;
                else
                {
                    Stream errstream = webResponse.GetResponseStream();
                    if (errstream != null)
                    {
                        StreamReader errreader = new StreamReader(errstream);
                        errsr = errreader.ReadToEnd();
                    }
                }
-----------------------------------------------------------------------------

Mar 2, 2009 at 9:20 PM
Note:   Sorry I had to remove ' + e.InnerException.Message' above as it has potential for another null reference if there is no inner exception and it was not populated at this point anyway.  I copied the above from my notes and not my live code.
Apr 29, 2009 at 4:05 PM
Another thing I would like to bring up: solr query "start" parameter calculation.

Currently QueryBuilder exposes Rows and Page properties and proceeds to calculate the start value behind the scenes as (Page - 1)*Rows. I think having an explicit Start property and allowing a client to set its value is a more flexible approach. I have a case where my pages are 10 rows long but I fetch 30 rows at a time (specific requirement) and on page 2 I want to start with row 11 but solrsharp logic makes it start at row 31.

So instead of setting a page I would always set the start (offset) property. I can provide a patch if needed.

May 12, 2009 at 9:48 PM

Another other thing, which i just noticed, is that specifying a facet limit changes facet sorting behavior!   So solrsharp needs a way to specify the desired sort per facet field:

  Solr1.2

facet.sort
This param determines the ordering of the facet field constraints.

•true - sort the constraints by count (highest count first)

•false - to return the constraints sorted in their index order (lexicographic by indexed term) . For terms in the ascii range, this will be alphabetically sorted.

 Solr1.4 -- the true/false values have been deprecated starting with Solr 1.4, instead use count for sorting by count, and index for sorting by index order.

The default is true/count if facet.limit is greater than 0, false/index otherwise.

This parameter can be specified on a per field basis.

Jul 29, 2009 at 10:12 PM

we set up solrsharp to handle multicore situations by expanding the solr config section as follows, and i can supply the code:

 

<solr>
        <server name="10017" mode="ReadWrite" url="http://iSS01.dmz.b.com:8983/solr/10017" />
        <server name="10018" mode="ReadWrite" url="http://iSS01.dmz.b.com:8983/solr/10018" />
        <server name="10019" mode="ReadWrite" url="http://iSS01.dmz.b.com:8983/solr/10019" />
        <server name="10020" mode="ReadWrite" url="http://iSS01.dmz.b.com:8983/solr/10020" />
        <server name="10021" mode="ReadWrite" url="http://iSS01.dmz.b.com:8983/solr/10021" />
</solr>

Dec 23, 2009 at 8:48 AM
robipete wrote:

we set up solrsharp to handle multicore situations by expanding the solr config section as follows, and i can supply the code:

 

<solr>
        <server name="10017" mode="ReadWrite" url="http://iSS01.dmz.b.com:8983/solr/10017" />
        <server name="10018" mode="ReadWrite" url="http://iSS01.dmz.b.com:8983/solr/10018" />
        <server name="10019" mode="ReadWrite" url="http://iSS01.dmz.b.com:8983/solr/10019" />
        <server name="10020" mode="ReadWrite" url="http://iSS01.dmz.b.com:8983/solr/10020" />
        <server name="10021" mode="ReadWrite" url="http://iSS01.dmz.b.com:8983/solr/10021" />
</solr>

Hi robipete,

I'm triyng to achieve that too... but i'm getting an error on line

'SolrSearcher solrSearcher = SolrSearchers.GetSearcher(Mode.Read);'

O have only 2 cores, both defined as you did in app.config.

Did you changed anything else?

 

Thanks in advance