My application generates logs in JSON format. The logs look like this:
{"log-level": "error", "day-time": "2013-06-21T11: 20: 17Z", "module": " The value of 'WS_VALIDATE' is the parameter '@LOGIN_ID', which was not given. "} Currently, I am pushing the above log line because it is in Mongodiibi but stores the datetime as Mongodiibi storage (which is expected). Now when I want to run some data crunching jobs on these logs, I prefer to store Daylight as the original ISOodetti data type of Mongo DB.
There are 3 ways in which I can think of doing this:
i) Parse each JSON log line and convert string into ISODAT type in application code and then Insert it. Cons: I have to parse every single row before pushing the Monggo DB, which will be a bit expensive
ii) Run a query to change the string date time of the last inserted document after each insert
element.Datetime = ISODate (element.Datetime) to ISODate; Cons: Again expensive, because I'm running an additional query to put the copy iii) Modify my logs at a mass point so that I do not need to parse the application code. Besides, just curious, can I configure to automatically convert the datile string into my own original isodate format?
TIA
Edit: I am using pymongo to insert json log
My file looks like this:
{"log-level": "error", "datetime": "2013-06-21T11: 20: 17Z", "module": "db", "method": "execute", "request" : "WS_VALIDATE", "Error": "Process or action 'WS_VALIDATE' expects the parameter '@LOGIN_ID', which was not supplied."}
There are hundreds of lines mentioned above, and this Is that how do i I'm putting in::
For the line in the logfile: collection.insert (json.loads (line)) For the line in the logfile: Data = json.loads (line) data ["Datetime"] = datetime.strptime (data ["Datetime"], "% Y -% M-% DTHH: MMZ") collection.insert (data) < / Pre>
What I want to do, I get rid of additional manipulation from time to time. Hope that clarifies this problem.
It seems that you already have the answer ... I have to live with it: logfile In the line for JSON Loads (line) data ["datetime"] = DTItim. URTM (Data ["Datetime"], "% Y-% M-% DTH: MMZ") collection.insert (data) I had a similar problem, but I already It did not know that I had a date time object, so I changed my JSON notification to something like this:
{"LogLevel": "error", "routine": {"__ timestamp__": "WC_VALIDATE", "Error": "Process or function 'WS_VALIDATE' parameter '", "DB", "Method": "Execution", "Request": "2013-06-21T11: 20: 17Z"} " Expects'LOGIN_ID ', which Was not provided. "} and json with it was parsed:
json.loads (data, object_hook = logHook) : If in '__timestamp__' d: return datetime.strptime (d ['__ timestamp__'], "% Y-% M-% DTHH: Defined in MMZ: " elif, elif with many other 'variables' to change it LogHook function can also be extended. Hope this helps!
No comments:
Post a Comment