如何根据python中JSON文件的最新时间戳条件创建和写入文件?

Mona • 5 年前 • 1348 次点击

这是我所拥有的大型JSON文件的一部分。我正在创建包含以下内容的文件夹 yamlPath 然后将内容从 sqlQuery . 但在重复“yamlname+yamlpath”的情况下,我的代码只创建具有第一个结果的文件,并插入 SQL查询 它从for循环中获取的内容。所以,在这个重复的场景中,我只想 SQL查询 并根据最新的/max创建文件 jobEndTimestamp .

如何使代码只提取最新的 作业结束时间戳 然后创建文件?

预期输出:

/app/computer/users/ship-notice-data.sql   -> select from table.b
/app/computer/a/users/boat-notice-data.sql -> select from table.b

{
"stream": [
{
  "applicationServiceId": "uhhj",
  "yamlName": "/users/ship-notice-data.yml",
  "yamlPath": "/app/computer",
  "jobStartTimestamp": "2018-09-15 04:12:46",
  "jobEndTimestamp": "2018-09-15 04:15:29",
  "sourceHostName": "Teradata",
  "sourceType": "Teradata",
  "targetHostName": "DB2",
  "targetType": "DB2",
  "sqlQuery": "select from table.a"
},{
  "applicationServiceId": "uhhj",
  "yamlName": "/users/ship-notice-data.yml",
  "yamlPath": "/app/computer",
  "jobStartTimestamp": "2018-09-15 21:12:46",
  "jobEndTimestamp": "2018-09-15 21:15:29",
  "sourceHostName": "Teradata",
  "sourceType": "Teradata",
  "targetHostName": "DB2",
  "targetType": "DB2",
  "sqlQuery": "select from table.b"
},{
  "applicationServiceId": "uhhj",
  "yamlName": "/users/car-notice-data.yaml",
  "yamlPath": "/app/computer/s",
  "jobStartTimestamp": "2018-09-15 04:12:46",
  "jobEndTimestamp": "2018-09-15 06:15:29",
  "sourceHostName": "Teradata",
  "sourceType": "Teradata",
  "targetHostName": "DB2",
  "targetType": "DB2",
  "sqlQuery": "select from table.b"
},{
  "applicationServiceId": "uhhj",
  "yamlName": "/users/boat-notice-data.yaml",
  "yamlPath": "/app/computer/a",
  "jobStartTimestamp": "2018-09-15 04:12:46",
  "jobEndTimestamp": "2018-09-15 06:15:29",
  "sourceHostName": "Teradata",
  "sourceType": "Teradata",
  "targetHostName": "DB2",
  "targetType": "DB2",
  "sqlQuery": "select from table.b"
}
]
}

这是我的代码:

with open('/Users/mona/stream.json', 'r') as f:
    item_dict = json.load(f)

for item in item_dict['stream']:
    if (item['applicationServiceId'] == 'uhhj' and 
            item.get('targetHostName') == 'DB2' and
            (item['targetType'] == 'DB2')):
        # print(item.get('applicationServiceId'))
        v3 = item.get('applicationServiceId')
        v4 = item.get('jobEndTimestamp')
        v = item.get('sqlQuery')
        v1 = item.get('yamlName')
        v2 = item.get('yamlPath')
        print(v1+v2+" "+v4+ " " +str(v))
        # v4 = str(item.get('yamlName').split('/')[-1].split('.')[0])
        # print(v4)

        originalPath = "/Users/mona/"

        fullPath = os.path.join(originalPath+v2+(v1.split('/')[1].split('/')[0])+'/'+(v1.split('/')[2].split('/')[0])+'/')
        # print(fullPath)
        os.makedirs(fullPath, mode=0o777, exist_ok=True)
        # print(v1)

        with open(fullPath + str(item.get('yamlName').split('/')[-1].split('.')[0]) + ".sql", "w") as newFile:
            newFile.write("%s \n" % (v))

Python社区是高质量的Python/Django开发社区
本文地址：http://www.python88.com/topic/38997

1348 次点击

文章 [ 2 ] | 最新文章 5 年前

• 1 楼

wpercy 6 年前

要对熊猫执行此操作,您可以使用数据创建一个数据帧,添加一个名为 fullPath 按那个分组,然后过滤到最大值 jobEndTimestamp 。像这样:

import pandas as pd

with open('/Users/mona/stream.json', 'r') as f:
    item_dict = json.load(f.read())

df = pd.DataFrame(item_dict['sources'])
df['fullPath'] = df['yamlPath'] + df['yamlName']
grouped = df.groupby('fullPath').apply(lambda d: d[d['jobEndTimestamp'] == d['jobEndTimestamp'].max()])

print grouped['sqlQuery']

会屈服的

fullPath
/app/computer/a/users/boat-notice-data.yaml  3    select from table.b
/app/computer/s/users/car-notice-data.yaml   2    select from table.b
/app/computer/users/ship-notice-data.yml     1    select from table.b
Name: sqlQuery, dtype: object

• 2 楼

ekiim 6 年前

也许这个解决方案是次优的,但它可以工作…

我们的想法是把字典按 yamlPath 以及每个路径,包含字典列表,其中 亚姆帕斯 .

然后,您拆分并重新加入路径,并保存名称(这样它就与您的操作系统兼容),最后,我将打开上下文,这样您就可以附加到刚刚打开的文件(在不存在的情况下创建,或者在不存在的情况下打开并附加,这就是为什么 'a' 参数)。

from collections import defaultdic
import os
objects = defaultdic(list)

for item in item_dict['stream']:
    objects[item.yamlPath].append(item)
for yamlPath in objects.keys():
    for item in yamlPath:
        temp = yamlPath.split("/") + item.yamlName.split("/")[:-1]
        name = item.yamlName.split("/")[-1]
        file_with_path = os.path.join(*temp)
        with open(os.path.join(*temp, name), 'a') as file:
            pass # Do what ever to the content

同样,这只是一个想法,它可能会起作用,(还没有尝试过),并且您在创建dirs时遇到困难,但是如果问题仍然是时间戳,请遵循与此相同的逻辑,并按所需的时间戳对它们进行分组,然后迭代 max(objects.keys()) (假设这些日期是可比的,如果不只是铸造它们的话)

登录后回复