Monday 15 July 2013

pymongo - Convert a ISODate string to mongoDB native ISODate data type -


My application generates logs in JSON format. The logs look like this:

  {"log-level": "error", "day-time": "2013-06-21T11: 20: 17Z", "module": " The value of 'WS_VALIDATE' is the parameter '@LOGIN_ID', which was not given. "}   

Currently, I am pushing the above log line because it is in Mongodiibi but stores the datetime as Mongodiibi storage (which is expected). Now when I want to run some data crunching jobs on these logs, I prefer to store Daylight as the original ISOodetti data type of Mongo DB.

There are 3 ways in which I can think of doing this:

i) Parse each JSON log line and convert string into ISODAT type in application code and then Insert it. Cons: I have to parse every single row before pushing the Monggo DB, which will be a bit expensive

ii) Run a query to change the string date time of the last inserted document after each insert

  element.Datetime = ISODate (element.Datetime) to ISODate; Cons: Again expensive, because I'm running an additional query to put the copy iii) Modify my logs at a mass point so that I do not need to parse the application code.  

Besides, just curious, can I configure to automatically convert the datile string into my own original isodate format?

TIA

Edit: I am using pymongo to insert json log

My file looks like this:

{"log-level": "error", "datetime": "2013-06-21T11: 20: 17Z", "module": "db", "method": "execute", "request" : "WS_VALIDATE", "Error": "Process or action 'WS_VALIDATE' expects the parameter '@LOGIN_ID', which was not supplied."}

There are hundreds of lines mentioned above, and this Is that how do i I'm putting in::

  For the line in the logfile: collection.insert (json.loads (line))   

For the line in the logfile: Data = json.loads (line) data ["Datetime"] = datetime.strptime (data ["Datetime"], "% Y -% M-% DTHH: MMZ") collection.insert (data) < / Pre>

What I want to do, I get rid of additional manipulation from time to time. Hope that clarifies this problem.

It seems that you already have the answer ... I have to live with it: logfile In the line for JSON Loads (line) data ["datetime"] = DTItim. URTM (Data ["Datetime"], "% Y-% M-% DTH: MMZ") collection.insert (data)

I had a similar problem, but I already It did not know that I had a date time object, so I changed my JSON notification to something like this:

  {"LogLevel": "error", "routine": {"__ timestamp__": "WC_VALIDATE", "Error": "Process or function 'WS_VALIDATE' parameter '", "DB", "Method": "Execution", "Request": "2013-06-21T11: 20: 17Z"} " Expects'LOGIN_ID ', which Was not provided. "}   

and json with it was parsed:

  json.loads (data, object_hook = logHook)   

: If in '__timestamp__' d: return datetime.strptime (d ['__ timestamp__'], "% Y-% M-% DTHH: Defined in MMZ:

  "  

elif, elif with many other 'variables' to change it LogHook function can also be extended.

Hope this helps!

No comments:

Post a Comment