Damn that was hard! Getting the DB up and running was easy, getting data into the db was easy but the map and reduce stuff wasn't easy to understand in the beginning... it got better :)
The Goal:
Evaluate technology and trying to glean a little bit of information on what kind of production we get from our machines from the log files our our application, PackNet.Server produces.
CouchDB usings javascript for it's map and reduce functions. Get ready to bang your head if you're new to js and MR (MapReduce).
So I found that using js for string manipulation of logs was more difficult than necessary. After some struggles some of the initial data was parsed prior to loading the data like pulling out machine id's. Another consideration is to try to make the key data that you will parse out to be consistently formatted. For example, writing regex to look for our serial numbers was painful because sometimes we were using SerialNo, Serial, Job, Serial Number, Order, etc etc all for the same thing. Try to add some consistency if you want to make your life easier.
The next thing I struggled with was getting data into the system. I used Hammock/Reflex-Net. This was super simple and I liked it but there wasn't support for bulk uploads. However the CouchDB rest endpoint does support it :D. There are some people online that had some issues w/ this interface and the size of data they were sending. I set my write data count to 1000 rows in the log file and never saw a problem.
Lessons:
getting data in and out was a snap and so far I have a couple databases in the 8gb size. I'm not seeing any kind of performance impacts yet. I'd use it but I also thought MongoDB was good. Just make a decision! :D
Resources:
The Goal:
Evaluate technology and trying to glean a little bit of information on what kind of production we get from our machines from the log files our our application, PackNet.Server produces.
CouchDB usings javascript for it's map and reduce functions. Get ready to bang your head if you're new to js and MR (MapReduce).
So I found that using js for string manipulation of logs was more difficult than necessary. After some struggles some of the initial data was parsed prior to loading the data like pulling out machine id's. Another consideration is to try to make the key data that you will parse out to be consistently formatted. For example, writing regex to look for our serial numbers was painful because sometimes we were using SerialNo, Serial, Job, Serial Number, Order, etc etc all for the same thing. Try to add some consistency if you want to make your life easier.
The next thing I struggled with was getting data into the system. I used Hammock/Reflex-Net. This was super simple and I liked it but there wasn't support for bulk uploads. However the CouchDB rest endpoint does support it :D. There are some people online that had some issues w/ this interface and the size of data they were sending. I set my write data count to 1000 rows in the log file and never saw a problem.
Lessons:
- Database size was a little more than twice the size of data on disk.
- Get familiar w/ js
- Get familiar with cur, wget or some other client that will allow you to get at the data. A web browser also works fine...
- When testing your MR or anything else start our or use a small database like 30MB with a sample of the data you want to test.
- If you can more easily pull out key information and store it in the document from a log than you could in js do it.
- MR when you want to reduce think about what the end result of the data will be and then create keys that will allow you to group and move up down the chain for more/less granular values.
- Example: Keys [<machineId>, <day>, <day w/ hour>, <day w/ hour and min>] this way we could easily see carton production for a machine by day, hour or minute; see design/view in sandbox code machine/machineCompletedJobs
getting data in and out was a snap and so far I have a couple databases in the 8gb size. I'm not seeing any kind of performance impacts yet. I'd use it but I also thought MongoDB was good. Just make a decision! :D
Resources:
- http://www.slideshare.net/okurow/couchdb-mapreduce-13321353
- http://wiki.apache.org/couchdb/HTTP_view_API
- <couchDBUrl:port>/_utils to get to your databases
- http://meyerweb.com/eric/tools/dencoder/ for an online html encoder
- https://wiki.apache.org/couchdb/HTTP_database_API
- C# Library (inactive but works) https://github.com/danielcrenna/hammock
- code (Packsize developers only) TFS/PackNet/Sandbox/dotServerLoadLoaderToCouchDB
