HaBo HaBo - 6 months ago 23
Bash Question

MongoDB shell script to search in all fields except a set of fields

All documents in a collection have mandatory set of common fields along with other custom fields which can vary in each document.

Example Document 1 :

{
id="",
Name:"",
Description:"",
Doc1Field1="",
Doc1Field2:"",
Doc1Field3:""
}


Example Document 2 :

{
id="",
Name:"",
Description:"",
Doc2Field1="",
Doc2Field2:"",
Doc2Field3:"",
Doc2Field4:""
}


As you can notice Doc1 and Doc2 have a set of fields in common and rest are different and both documents are in one collection.

Now I need to write a shell script, that can search for a text in all fields other than the common fields.

I would like to list the common fields, don't want to do any comparison. In other words, search for a text in all fields in collections except a set of fields

I tried using text search indexing, but not working out.

db.MyCollection.createIndex({'$**': 'text'}, {name: 'FullTextIndex'})

db.MyCollection.find({$text: {$search: 'myWord'}}).count()


posted that question here MongoDB full text search

Answer

This is how I ended up to find a solution. Get a list of common fields, then for each project exclude these common fields and make list of other fields. using that list construct a query.

public virtual Dictionary<int, List<string>> GetOtherFields()
        {
            List<OtherField> otherFields = new List<OtherField>();
            List<Project> projects;
            using (myEntities context = new myEntities())
            {
                projects = context.Projects.ToList();
            }
            var commonFields = GetCommonFields();
            foreach (var project in projects)
            {
                var otherFieldsOfAProject = GetOtherFieldsByProjectId(project.Id, defaultFields);
                if (otherFieldsOfAProject.Count > 0)
                    otherFields.AddRange(otherFieldsOfAProject);
            }


            return OtherFieldsToDictionary(otherFields);
        }


 public virtual List<OtherField> GetOtherFieldsByProjectId(int projectId, List<string> commonFields = null)
        {
            if (commonFields  == null)
                commonFields  = GetCommonFields();
            Assembly assembly = Assembly.Load("NameSpace");
            var myType = assembly.GetType($"NameSpace.Imp{projectId}", false);
            if (myType != null)
            {
                var properties = myType.GetProperties().ToList();

                var otherFieldsOfAProject = properties.Where(
                    property =>
                        !commonFields.Any(
                            x => property.Name.Equals(x, StringComparison.InvariantCultureIgnoreCase))
                        && property.PropertyType.Name == typeof(string).Name)
                    .Select(x => new OtherField { ProjectId = projectId, FieldName = x.Name }).ToList();

                return otherFieldsOfAProject;
            }
            return new List<OtherField>();
        }


private static Dictionary<int, List<string>> OtherFieldsToDictionary(List<OtherField> otherFields)
        {
            var queryDictionary = otherFields.GroupBy(x => x.ProjectId)
                .ToDictionary(g => g.Key, g => g.Select(y => y.FieldName).ToList());
            return queryDictionary;
        }