Refactoring and other resources

DRY can drown you.  Great talk on refactoring and object oriented basics
https://www.youtube.com/watch?v=8bZh5LMaSmE

by Sandi Metz

Lessons Learned and Initial Thoughts on CouchDB using C# to upload data

Damn that was hard!  Getting the DB up and running was easy, getting data into the db was easy but the map and reduce stuff wasn't easy to understand in the beginning...  it got better :)

The Goal:
Evaluate technology and trying to glean a little bit of information on what kind of production we get from our machines from the log files our our application, PackNet.Server produces.

CouchDB usings javascript for it's map and reduce functions.  Get ready to bang your head if you're new to js and MR (MapReduce).

So I found that using js for string manipulation of logs was more difficult than necessary.  After some struggles some of the initial data was parsed prior to loading the data like pulling out machine id's.  Another consideration is to try to make the key data that you will parse out to be consistently formatted.  For example, writing regex to look for our serial numbers was painful because sometimes we were using SerialNo, Serial, Job, Serial Number, Order, etc etc all for the same thing.  Try to add some consistency if you want to make your life easier.

The next thing I struggled with was getting data into the system.  I used Hammock/Reflex-Net.  This was super simple and I liked it but there wasn't support for bulk uploads.  However the CouchDB rest endpoint does support it :D.  There are some people online that had some issues w/ this interface and the size of data they were sending.  I set my write data count to 1000 rows in the log file and never saw a problem.

Lessons:

• Database size was a little more than twice the size of data on disk.  
• Get familiar w/ js
• Get familiar with cur, wget or some other client that will allow you to get at the data.  A web browser also works fine...
• When testing your MR or anything else start our or use a small database like 30MB with a sample of the data you want to test.
• If you can more easily pull out key information and store it in the document from a log than you could in js do it.
• MR when you want to reduce think about what the end result of the data will be and then create keys that will allow you to group and move up down the chain for more/less granular values.
• Example: Keys [<machineId>, <day>, <day w/ hour>, <day w/ hour and min>]  this way we could easily see carton production for a machine by day, hour or minute;  see design/view in sandbox code machine/machineCompletedJobs

Final Thoughts:
getting data in and out was a snap and so far I have a couple databases in the 8gb size.  I'm not seeing any kind of performance impacts yet.  I'd use it but I also thought MongoDB was good.  Just make a decision!  :D

Resources:

• http://www.slideshare.net/okurow/couchdb-mapreduce-13321353
• http://wiki.apache.org/couchdb/HTTP_view_API
• <couchDBUrl:port>/_utils to get to your databases
• http://meyerweb.com/eric/tools/dencoder/ for an online html encoder
• https://wiki.apache.org/couchdb/HTTP_database_API
• C# Library (inactive but works) https://github.com/danielcrenna/hammock
• code (Packsize developers only) TFS/PackNet/Sandbox/dotServerLoadLoaderToCouchDB

WCF Delay when making multiple calls

We were having an issue with a delay that would only surface occasionally. I tracked it down to a service we have that holds the connection open until it has something to send to the client or 15 seconds and it sends a heartbeat back.
Every so often things would stop responding. I tracked it down to be occurring only while we were waiting for the blocking call to return. This did not happen all the time but we could repo it regularly by using the app.

I did a google search as always and found a wcf setting called MaxConnections. So I put it in and changed the value to 100. Tested it and voila!

We had some helper classes that would open connections and keep the services alive (inContact.WCFConnector)

http://msdn.microsoft.com/en-us/library/wcf.maxconnections(v=bts.10).aspx

system.net>>>connectionManagement>>>add address = "*" maxconnection = "100" />

RegEx for Dates including leap year with locale support

First of all when you are playing with regex it helps to have a tool that will let you see what is happening. For this I have found Expresso (http://www.ultrapico.com/Expresso.htm found here in 2011 and is a free tool).

While working on the localization of our entire suite of systems & applications we needed a regular expression that would validate user input for a date. So depending on the user locale/language the date format changes. We needed three: Day first(Europe) DD.MM.YYYY, Month first MM/DD/YYYY (US) and year first YYYY/MM/DD (chinese).

So we set the value of the Threads current context and used the DateFormatInfo class from there to determine which format to use.
Then we found a regEx that worked for US (ref.1) and modified it for out use.

^(?

(?

(?0?[1-9]|1[012])

[- /.]{1} #seperator

(?0?[1-9]|1\d|2[0-8])

|

(?0?[13456789]|1[012])

[- /.]{1} #seperator

(?29|30)

|

(?0?[13578]|1[02])

[- /.]{1} #seperator

(?31))

[- /.]{1} #seperator

(?19|[2-9]\d)

(?\d{2})

|

(?0?2)

[- /.]{1} #seperator

(?29)

[- /.]{1} #seperator

(?(?19|[2-9]\d)(?0[48]|[2468][048]|[13579][26])|(?(?[2468][048]|[3579][26])00)) # leap year 02 or 2 / 29

)$

|

#German

^(?

(?

(?0?[1-9]|1\d|2[0-8])

[- /.]{1} #seperator

(?0?[1-9]|1[012])

|

(?29|30)

[- /.]{1} #seperator

(?0?[13456789]|1[012])

|

(?31)

[- /.]{1} #seperator

(?0?[13578]|1[02])

)

[- /.]{1} #seperator

(?19|[2-9]\d)

(?\d{2})

|

(?29)

[- /.]{1} #seperator

(?0?2)

[- /.]{1} #seperator

(?

(?19|[2-9]\d)

(?0[48]|[2468][048]|[13579][26])

|

(?

(?[2468][048]|[3579][26])00)) # leap year 02 or 2 / 29

)$

|

#CHN

^(?

(?19|[2-9]\d)

(?\d{2})

[- /.]{1} #seperator

(?

(?0?[1-9]|1[012])

[- /.]{1} #seperator

(?0?[1-9]|1\d|2[0-8])

|

(?0?[13456789]|1[012])

[- /.]{1} #seperator

(?29|30)

|

(?0?[13578]|1[02])

[- /.]{1} #seperator

(?31))

|

(?(?19|[2-9]\d)(?0[48]|[2468][048]|[13579][26])|(?(?[2468][048]|[3579][26])00)) # leap year 02 or 2 / 29

[- /.]{1} #seperator

(?0?2)

[- /.]{1} #seperator

(?29)

)$

-------------------------------
Ref.1 - by Dany Lauener; http://regexlib.com/UserPatterns.aspx?authorId=81355952-f53d-4142-bc5c-aab2beae19f3; search for "MM/dd/yyyy with 100% leap years. Valid since year 1900"

VS Remote debugging OMG

It really shouldn't be this difficult... I spent a couple days trying to get this running and finally, boom it's on!
The remote machine did not have a firewall running for us.
Start the debugger on the remote machine, you will see the name and who it is running as.
http://msdn.microsoft.com/en-us/library/ee126350.aspx
We had rules set up for inbound and outbound specifically for Devenv.exe after removing all rules then following the above like things magically worked.

Note: When connecting to the remote machine w/ VS make sure you enter credentials if necessary like /username@machine eg ucn/rob.ling@eng-ngpcl07

Problems we had: remote debugger would show that the connection was being made but then our VS connection would time out. I think this was fixed by adding the UDP rule.

And this is fun.
http://www.youtube.com/watch?v=sSUXTFceilo

Tasks and anonymous delegates in for loops

Today at work I found a problem with using annonamous delagates tasks and foreach. Below is the code

            List waitingSkills = new List();
            foreach (var si in AgentSkills)
            {
                var t = Task.Factory.StartNew(delegate
                                               {
                                                   try
                                                   {
                                                       si.Value.Dispositions = GetDispositions(si.Value.SkillNo);
                                                   }
                                                   catch (Exception ex)
                                                   {  ...
                                                   }
                                                  
                                               });
               
                waitingSkills.Add(t);
            }

            var done = false;

            while (!done)
            {
                done = true;
                for (int i = 0; i < waitingSkills.Count; i++)
                {
                    if (!waitingSkills[i].IsCompleted)
                    {
                        done = false;
                        Thread.Sleep(100);
                        break;
                    }

                }

            }

---------------------------------
The problem is that the value of si is the local value or last value assigned before the task kicks off. In my case the value was always the last value in the AgentSkills collection.

Here is the refactor. I got there by using this example found in Introducing Visual C# 2010

public void ProcessUpdateSkillsEvent(UpdateAgentSkillsEvent e)
        {
            ReskillSkills = new Dictionary();
            AgentSkills = new Dictionary();

            e.ReskillSkills.Where(r => r.SkillType == "Inbound").ToList().ForEach(rs => ReskillSkills.Add(rs.ID, ConvertAgentSkillToISkillItem(rs)));
            e.Skills.ToList().ForEach(rs => AgentSkills.Add(rs.ID, ConvertAgentSkillToISkillItem(rs)));

            var listOfSkills = AgentSkills.Values.ToList();
            var waitingSkills = new Task[AgentSkills.Count];

            //Create a func so we can pass the skillItem to the method so we can load the dispositions in a seperate thread.
            Func funcGetDispositions = FuncGetDispositions;
            for (int i = 0; i < listOfSkills.Count; ++i)
            {
                waitingSkills[i] = Task.Factory.StartNew(funcGetDispositions, listOfSkills[i]);
            }

            Task.WaitAll(waitingSkills.ToArray());

            if (OnAgentSkillsUpdated != null)
                OnAgentSkillsUpdated(AgentSkills.Values.ToList());

            if (OnReskillSkillsUpdated != null)
                OnReskillSkillsUpdated(ReskillSkills.Values.ToList());
        }

        public SkillItem FuncGetDispositions(object obj)
        {
            var skill = (SkillItem) obj;
            try
            {
                skill.Dispositions = GetDispositions(skill.SkillNo);
            }
            catch (Exception ex)
            {...
            }
            return skill;

        }

Moq Setting local variable using callbacks.

http://code.google.com/p/moq/wiki/QuickStart

// Set up the mock:

AgentLoginParams passedObject = null;

moduleMock.Setup(mm=>mm.Login(It.IsAny())).Callback(alp=> passedObject = alp);

// Do something you’re testing

controller.Login(station, userName, password);

// Assert that something was called with the correct data

Assert.IsNotNull(passedObject);

Assert.AreEqual(userName,passedObject.UserName);