新建
Baseline Notebook for AI4SCup-LLMKG (0) —— 数据预处理与标准化
Judy Lin
推荐镜像 :ai4s-cup-metrics:0.3
推荐机型 :c2_m4_cpu
赞
11
36
目录
数据集
赛数据集(v8)
©️ Copyright 2023 @ Authors
Author:
Mujie Lin 📨, Yongge Li 📨
Date: 2023-02
共享协议：本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可。
由于不同大模型对训练用的json数据格式要求不同，本notebook的预处理仅供用于Llama2-7B,14B+Qlora微调，处理后每个条目格式如下：
{"instruction": " [Your Prompt]",
"input": "[文献摘要]",
"output": "[提取出的二元组或三元组]"}
代码
文本
[1]
import os
DATA_PATH = os.getenv('DATA_PATH')
# If DATA_PATH is not set, assign a default value and raise a warning
if not DATA_PATH:
DATA_PATH = '/bohr/AGAC-GDA-0ifh/v8/'
print("Warning: DATA_PATH environment variable is not set. Using default path:", DATA_PATH)
代码
文本
[2]
import json
import pandas as pd
import os
代码
文本
处理后的数据全部保存到与notebook同目录的data.json
代码
文本
[3]
new_json_file_path = "data.json"
new_data = []
代码
文本
任务一
代码
文本
[6]
!pip install openpyxl
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com Collecting openpyxl Downloading openpyxl-3.1.2-py2.py3-none-any.whl (249 kB) |████████████████████████████████| 249 kB 311 kB/s eta 0:00:01 Collecting et-xmlfile Downloading et_xmlfile-1.1.0-py3-none-any.whl (4.7 kB) Installing collected packages: et-xmlfile, openpyxl Successfully installed et-xmlfile-1.1.0 openpyxl-3.1.2 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
代码
文本
[7]
# 定义处理单个JSON文件的函数
def process_json_file_task1(file_path, triad_df):
with open(file_path, 'r', encoding='utf-8') as file:
data = json.load(file)
input_text = data['text']
sourceid = data['sourceid']
# 查找匹配的三元组
matching_rows = triad_df[triad_df['PMID'] == int(sourceid)]
output = ', '.join(f"({row['GENE']}, {row['FUNCTION']}, {row['DISEASE']})" for _, row in matching_rows.iterrows())
return {
## "instruction"为自己定义的prompt
"instruction": "In this Gene-Disease relation extraction task, you need to follow 3 steps. You need to extract the (gene, function change, disease) triplet from the text, such as: (SHROOM3, LOF, Neural tube defects). The second element in the triple means the regulation that the gene produces to the disease. Types of regulations are: LOF and GOF, which indicate loss or gain of function; REG, which indicates a general regulatory relationship; COM, which indicates that the functional change between genes and diseases is more complex, and it is difficult to determine whether the functional change is LOF or GOF. Please return all the relations extracted from the text in ternary format (GENE, FUNCTION, DISEASE). If there are more than one triple, please write in this form: '(GENE, FUNCTION, DISEASE),(GENE, FUNCTION, DISEASE),......'.",
"input": input_text,
"output": output
}
代码
文本
[8]
triad_df = pd.read_excel(DATA_PATH+'task1/train_triad.xlsx')
json_files_directory = DATA_PATH +'task1/'
# 处理所有文件并收集数据
for file_name in os.listdir(json_files_directory):
if file_name.endswith('.json'):
try:
file_path = os.path.join(json_files_directory, file_name)
result = process_json_file_task1(file_path, triad_df)
new_data.append(result)
print(f"Transformation complete for {file_name}")
except:
continue
Transformation complete for PubMed-15678000.json Transformation complete for PubMed-15773749.json Transformation complete for PubMed-15773758.json Transformation complete for PubMed-15857086.json Transformation complete for PubMed-15880705.json Transformation complete for PubMed-15931689.json Transformation complete for PubMed-15951859.json Transformation complete for PubMed-16087292.json Transformation complete for PubMed-16087766.json Transformation complete for PubMed-16126871.json Transformation complete for PubMed-16134171.json Transformation complete for PubMed-16151697.json Transformation complete for PubMed-16378925.json Transformation complete for PubMed-16447768.json Transformation complete for PubMed-16447769.json Transformation complete for PubMed-16505158.json Transformation complete for PubMed-16541399.json Transformation complete for PubMed-16720617.json Transformation complete for PubMed-16730748.json Transformation complete for PubMed-16770791.json Transformation complete for PubMed-16799052.json Transformation complete for PubMed-16891820.json Transformation complete for PubMed-16909393.json Transformation complete for PubMed-16941474.json Transformation complete for PubMed-17076268.json Transformation complete for PubMed-17111025.json Transformation complete for PubMed-17277857.json Transformation complete for PubMed-17347989.json Transformation complete for PubMed-17408421.json Transformation complete for PubMed-17417636.json Transformation complete for PubMed-17449919.json Transformation complete for PubMed-17551331.json Transformation complete for PubMed-17880458.json Transformation complete for PubMed-17922155.json Transformation complete for PubMed-17922851.json Transformation complete for PubMed-17998451.json Transformation complete for PubMed-18033041.json Transformation complete for PubMed-18176357.json Transformation complete for PubMed-18230725.json Transformation complete for PubMed-18285826.json Transformation complete for PubMed-18317597.json Transformation complete for PubMed-18368133.json Transformation complete for PubMed-18388506.json Transformation complete for PubMed-18393234.json Transformation complete for PubMed-18653336.json Transformation complete for PubMed-18791947.json Transformation complete for PubMed-18938114.json Transformation complete for PubMed-19085932.json Transformation complete for PubMed-19108832.json Transformation complete for PubMed-19137020.json Transformation complete for PubMed-19169481.json Transformation complete for PubMed-19245173.json Transformation complete for PubMed-19288050.json Transformation complete for PubMed-19347998.json Transformation complete for PubMed-19358333.json Transformation complete for PubMed-19367640.json Transformation complete for PubMed-19373257.json Transformation complete for PubMed-19394700.json Transformation complete for PubMed-19401457.json Transformation complete for PubMed-19438860.json Transformation complete for PubMed-19449169.json Transformation complete for PubMed-19473408.json Transformation complete for PubMed-19657723.json Transformation complete for PubMed-19755381.json Transformation complete for PubMed-19863546.json Transformation complete for PubMed-19919001.json Transformation complete for PubMed-20015893.json Transformation complete for PubMed-20031356.json Transformation complete for PubMed-20116044.json Transformation complete for PubMed-20132240.json Transformation complete for PubMed-20152552.json Transformation complete for PubMed-20207543.json Transformation complete for PubMed-20222934.json Transformation complete for PubMed-20236351.json Transformation complete for PubMed-20238024.json Transformation complete for PubMed-21868336.json Transformation complete for PubMed-22001500.json Transformation complete for PubMed-22340355.json Transformation complete for PubMed-22483671.json Transformation complete for PubMed-22508176.json Transformation complete for PubMed-22539353.json Transformation complete for PubMed-22541666.json Transformation complete for PubMed-22666734.json Transformation complete for PubMed-22688420.json Transformation complete for PubMed-22727569.json Transformation complete for PubMed-22818872.json Transformation complete for PubMed-22916206.json Transformation complete for PubMed-23023219.json Transformation complete for PubMed-23072184.json Transformation complete for PubMed-23247051.json Transformation complete for PubMed-23271757.json Transformation complete for PubMed-23329183.json Transformation complete for PubMed-23443021.json Transformation complete for PubMed-23506586.json Transformation complete for PubMed-23541687.json Transformation complete for PubMed-23719190.json Transformation complete for PubMed-23727441.json Transformation complete for PubMed-23761384.json Transformation complete for PubMed-23782367.json Transformation complete for PubMed-23791648.json Transformation complete for PubMed-23892522.json Transformation complete for PubMed-24075313.json Transformation complete for PubMed-24488655.json Transformation complete for PubMed-24655737.json Transformation complete for PubMed-24659738.json Transformation complete for PubMed-24715752.json Transformation complete for PubMed-24771548.json Transformation complete for PubMed-24781643.json Transformation complete for PubMed-24885126.json Transformation complete for PubMed-24909267.json Transformation complete for PubMed-25192508.json Transformation complete for PubMed-25233259.json Transformation complete for PubMed-25353698.json Transformation complete for PubMed-25381065.json Transformation complete for PubMed-25439737.json Transformation complete for PubMed-25597551.json Transformation complete for PubMed-25604326.json Transformation complete for PubMed-25854144.json Transformation complete for PubMed-25993682.json Transformation complete for PubMed-26018748.json Transformation complete for PubMed-26076881.json Transformation complete for PubMed-26083983.json Transformation complete for PubMed-26094131.json Transformation complete for PubMed-26207643.json Transformation complete for PubMed-26260218.json Transformation complete for PubMed-26471493.json Transformation complete for PubMed-26545172.json Transformation complete for PubMed-26564087.json Transformation complete for PubMed-26564090.json Transformation complete for PubMed-26572961.json Transformation complete for PubMed-26624524.json Transformation complete for PubMed-26662040.json Transformation complete for PubMed-26671124.json Transformation complete for PubMed-26677118.json Transformation complete for PubMed-26690517.json Transformation complete for PubMed-26706855.json Transformation complete for PubMed-26707870.json Transformation complete for PubMed-26742514.json Transformation complete for PubMed-26791972.json Transformation complete for PubMed-26940370.json Transformation complete for PubMed-26974330.json Transformation complete for PubMed-26990546.json Transformation complete for PubMed-27070432.json Transformation complete for PubMed-27095120.json Transformation complete for PubMed-27106217.json Transformation complete for PubMed-27120018.json Transformation complete for PubMed-27127178.json Transformation complete for PubMed-27149098.json Transformation complete for PubMed-27173951.json Transformation complete for PubMed-27206675.json Transformation complete for PubMed-27225717.json Transformation complete for PubMed-27235476.json Transformation complete for PubMed-27241276.json Transformation complete for PubMed-27241678.json Transformation complete for PubMed-27265460.json Transformation complete for PubMed-27270313.json Transformation complete for PubMed-27292088.json Transformation complete for PubMed-27307693.json Transformation complete for PubMed-27323706.json Transformation complete for PubMed-27376611.json Transformation complete for PubMed-27379765.json Transformation complete for PubMed-27432949.json Transformation complete for PubMed-27432961.json Transformation complete for PubMed-27487919.json Transformation complete for PubMed-27585885.json Transformation complete for PubMed-27604558.json Transformation complete for PubMed-27617348.json Transformation complete for PubMed-27647186.json Transformation complete for PubMed-27647909.json Transformation complete for PubMed-27647924.json Transformation complete for PubMed-27654141.json Transformation complete for PubMed-27663151.json Transformation complete for PubMed-27699788.json Transformation complete for PubMed-27708270.json Transformation complete for PubMed-27734835.json Transformation complete for PubMed-27747153.json Transformation complete for PubMed-27774737.json Transformation complete for PubMed-27863268.json Transformation complete for PubMed-27874073.json Transformation complete for PubMed-27878139.json Transformation complete for PubMed-27882351.json Transformation complete for PubMed-27900366.json Transformation complete for PubMed-28052128.json Transformation complete for PubMed-28054739.json Transformation complete for PubMed-28063748.json Transformation complete for PubMed-28081536.json Transformation complete for PubMed-28100473.json Transformation complete for PubMed-28128203.json Transformation complete for PubMed-28150888.json Transformation complete for PubMed-28151491.json Transformation complete for PubMed-28163187.json Transformation complete for PubMed-28183324.json Transformation complete for PubMed-28191891.json Transformation complete for PubMed-28211240.json Transformation complete for PubMed-28297138.json Transformation complete for PubMed-28317092.json Transformation complete for PubMed-28389584.json Transformation complete for PubMed-28390992.json Transformation complete for PubMed-28423230.json Transformation complete for PubMed-28425595.json Transformation complete for PubMed-28477109.json Transformation complete for PubMed-28572275.json Transformation complete for PubMed-28590056.json Transformation complete for PubMed-28640966.json Transformation complete for PubMed-28748849.json Transformation complete for PubMed-28777843.json Transformation complete for PubMed-28859693.json Transformation complete for PubMed-28877480.json Transformation complete for PubMed-28972032.json Transformation complete for PubMed-29088233.json Transformation complete for PubMed-29159838.json Transformation complete for PubMed-29220510.json Transformation complete for PubMed-29253601.json Transformation complete for PubMed-29405930.json Transformation complete for PubMed-29578119.json Transformation complete for PubMed-29892959.json Transformation complete for PubMed-29957067.json Transformation complete for PubMed-29973161.json Transformation complete for PubMed-30031150.json Transformation complete for PubMed-30036279.json Transformation complete for PubMed-30081849.json Transformation complete for PubMed-30128655.json Transformation complete for PubMed-30144815.json Transformation complete for PubMed-30454682.json Transformation complete for PubMed-30702381.json
代码
文本
任务二
代码
文本
[9]
def list2str(relations):
return ', '.join(f"({relation[0]},{relation[1]})" for relation in relations)
def list2str_entity(entity_list):
return ', '.join(entity for entity in entity_list)
# Load the original data
with open(DATA_PATH+'task2/train.json', 'r') as file:
data = json.load(file)
for key, value in data.items():
abstract = value['abstract']
chemical_names = list(value['chemical2id'].keys())
disease_names = list(value['disease2id'].keys())
relations = []
for rel in value['relations']:
chemical_id = rel['chemical']
disease_id = rel['disease']
# Find the corresponding names
chemical_name = [name for name, id in value['chemical2id'].items() if id == chemical_id]
disease_name = [name for name, id in value['disease2id'].items() if id == disease_id]
if chemical_name and disease_name:
relations.append((chemical_name[0], disease_name[0]))
transformed_entry = {
"instruction": "You are a biologist AI. I'll give you the abstract of literature. Please identify all the [compound,disease] relations in the abstract, and just give me a list of all relations you recognized. To be mentioned, all the relations must be strictly presented to me in the format [[compound 1, disease 1],[compound 2, disease 2],[...,...],...], without adding any additional comments or explanations!",
"input": abstract,
"response": list2str(relations)
}
print(f"Transformation complete for {key}")
new_data.append(transformed_entry)
Transformation complete for 227508 Transformation complete for 354896 Transformation complete for 435349 Transformation complete for 603022 Transformation complete for 1378968 Transformation complete for 1420741 Transformation complete for 1601297 Transformation complete for 1967484 Transformation complete for 2234245 Transformation complete for 2385256 Transformation complete for 2505783 Transformation complete for 2515254 Transformation complete for 2572625 Transformation complete for 2632720 Transformation complete for 2670794 Transformation complete for 2696505 Transformation complete for 2924746 Transformation complete for 2951327 Transformation complete for 3192036 Transformation complete for 3409645 Transformation complete for 3412544 Transformation complete for 3425586 Transformation complete for 3437726 Transformation complete for 3693336 Transformation complete for 3780846 Transformation complete for 3800626 Transformation complete for 3827439 Transformation complete for 3997294 Transformation complete for 4071154 Transformation complete for 6103707 Transformation complete for 6229975 Transformation complete for 6286738 Transformation complete for 6287825 Transformation complete for 6386793 Transformation complete for 6387529 Transformation complete for 6692345 Transformation complete for 6773726 Transformation complete for 6888657 Transformation complete for 7265370 Transformation complete for 7423039 Transformation complete for 7444978 Transformation complete for 7834920 Transformation complete for 7881871 Transformation complete for 7930386 Transformation complete for 7988234 Transformation complete for 8073369 Transformation complete for 8302922 Transformation complete for 8410052 Transformation complete for 8423889 Transformation complete for 8682684 Transformation complete for 9625142 Transformation complete for 9766615 Transformation complete for 10193204 Transformation complete for 10526274 Transformation complete for 10669626 Transformation complete for 11379838 Transformation complete for 11419773 Transformation complete for 11581460 Transformation complete for 11706060 Transformation complete for 11752354 Transformation complete for 12369736 Transformation complete for 12639165 Transformation complete for 12653683 Transformation complete for 14659530 Transformation complete for 15804801 Transformation complete for 16160878 Transformation complete for 16274958 Transformation complete for 16330766 Transformation complete for 16574712 Transformation complete for 17111419 Transformation complete for 17175308 Transformation complete for 17244258 Transformation complete for 18020536 Transformation complete for 18023325 Transformation complete for 18208574 Transformation complete for 18343374 Transformation complete for 18417364 Transformation complete for 18442015 Transformation complete for 18619688 Transformation complete for 19308880 Transformation complete for 19631624 Transformation complete for 20003049 Transformation complete for 20196116 Transformation complete for 20470218 Transformation complete for 20722491 Transformation complete for 20882060 Transformation complete for 26094 Transformation complete for 322550 Transformation complete for 869641 Transformation complete for 1749407 Transformation complete for 1786266 Transformation complete for 1835291 Transformation complete for 1919871 Transformation complete for 1987816 Transformation complete for 2054792 Transformation complete for 2304736 Transformation complete for 2312209 Transformation complete for 2549018 Transformation complete for 2611118 Transformation complete for 2673163 Transformation complete for 3107448 Transformation complete for 3341566 Transformation complete for 3564823 Transformation complete for 3676049 Transformation complete for 3719553 Transformation complete for 4008111 Transformation complete for 6133211 Transformation complete for 6503301 Transformation complete for 6727060 Transformation complete for 7083920 Transformation complete for 7269015 Transformation complete for 7352670 Transformation complete for 7504976 Transformation complete for 7628595 Transformation complete for 7858459 Transformation complete for 7862923 Transformation complete for 7919560 Transformation complete for 8092427 Transformation complete for 8638206 Transformation complete for 8669433 Transformation complete for 8690168 Transformation complete for 8739323 Transformation complete for 8752018 Transformation complete for 9390208 Transformation complete for 9406968 Transformation complete for 9587734 Transformation complete for 9698967 Transformation complete for 9855119 Transformation complete for 9869257 Transformation complete for 10342929 Transformation complete for 10457883 Transformation complete for 10739826 Transformation complete for 10791295 Transformation complete for 11147747 Transformation complete for 11198499 Transformation complete for 11391224 Transformation complete for 11426838 Transformation complete for 11569530 Transformation complete for 11587867 Transformation complete for 11679859 Transformation complete for 12041669 Transformation complete for 12198388 Transformation complete for 12574103 Transformation complete for 12615818 Transformation complete for 12789195 Transformation complete for 12820454 Transformation complete for 14657095 Transformation complete for 14765563 Transformation complete for 15036754 Transformation complete for 15145918 Transformation complete for 15233872 Transformation complete for 15458908 Transformation complete for 15737522 Transformation complete for 16005948 Transformation complete for 16167916 Transformation complete for 16403073 Transformation complete for 16755009 Transformation complete for 16904497 Transformation complete for 16938416 Transformation complete for 17147461 Transformation complete for 17241784 Transformation complete for 17261653 Transformation complete for 17343925 Transformation complete for 17400887 Transformation complete for 17612891 Transformation complete for 18081909 Transformation complete for 18083142 Transformation complete for 18182964 Transformation complete for 18217897 Transformation complete for 18996674 Transformation complete for 19058010 Transformation complete for 19581773 Transformation complete for 19759529 Transformation complete for 19957053 Transformation complete for 20619828 Transformation complete for 891050 Transformation complete for 983936 Transformation complete for 1428568 Transformation complete for 1720453 Transformation complete for 1833784 Transformation complete for 1867351 Transformation complete for 2070391 Transformation complete for 2266990 Transformation complete for 2348231 Transformation complete for 2355241 Transformation complete for 2445283 Transformation complete for 2950248 Transformation complete for 3015567 Transformation complete for 3031535 Transformation complete for 3125768 Transformation complete for 3187073 Transformation complete for 3371379 Transformation complete for 3503576 Transformation complete for 3560095 Transformation complete for 3714122 Transformation complete for 3828020 Transformation complete for 4812392 Transformation complete for 6518066 Transformation complete for 6534871 Transformation complete for 6538499 Transformation complete for 6747681 Transformation complete for 6861444 Transformation complete for 7053303 Transformation complete for 7088431 Transformation complete for 7248170 Transformation complete for 7453952 Transformation complete for 7802851 Transformation complete for 8319760 Transformation complete for 8386779 Transformation complete for 8421099 Transformation complete for 8649546 Transformation complete for 8919272 Transformation complete for 9199746 Transformation complete for 9249847 Transformation complete for 9284778 Transformation complete for 9522143 Transformation complete for 9522152 Transformation complete for 9672936 Transformation complete for 9721172 Transformation complete for 9727773 Transformation complete for 9754849 Transformation complete for 10365197 Transformation complete for 10704919 Transformation complete for 10706004 Transformation complete for 10721819 Transformation complete for 10737864 Transformation complete for 10743446 Transformation complete for 11007689 Transformation complete for 11256525 Transformation complete for 11334364 Transformation complete for 11642480 Transformation complete for 12063090 Transformation complete for 12084448 Transformation complete for 12684739 Transformation complete for 12716030 Transformation complete for 12757899 Transformation complete for 12905102 Transformation complete for 14513889 Transformation complete for 15009014 Transformation complete for 15120741 Transformation complete for 15275829 Transformation complete for 15602202 Transformation complete for 16181582 Transformation complete for 16298782 Transformation complete for 16428221 Transformation complete for 16858720 Transformation complete for 17466854 Transformation complete for 17562951 Transformation complete for 17600377 Transformation complete for 17639754 Transformation complete for 17854040 Transformation complete for 18221780 Transformation complete for 18261172 Transformation complete for 18329269 Transformation complete for 18410508 Transformation complete for 18503483 Transformation complete for 18560792 Transformation complete for 18726058 Transformation complete for 18754075 Transformation complete for 18768591 Transformation complete for 19299179 Transformation complete for 19356053 Transformation complete for 19729346 Transformation complete for 20080983 Transformation complete for 20520283 Transformation complete for 20588063 Transformation complete for 19820426 Transformation complete for 1616457 Transformation complete for 567256 Transformation complete for 17439425 Transformation complete for 9351491 Transformation complete for 18752389 Transformation complete for 20098969 Transformation complete for 9653867 Transformation complete for 17608141 Transformation complete for 2559236 Transformation complete for 21029050 Transformation complete for 10901305 Transformation complete for 19037603 Transformation complete for 20084309 Transformation complete for 8231633 Transformation complete for 10091617 Transformation complete for 19269743 Transformation complete for 3070035 Transformation complete for 1147734 Transformation complete for 12202650 Transformation complete for 18951540 Transformation complete for 19657887 Transformation complete for 9660111 Transformation complete for 11431197 Transformation complete for 7449470 Transformation complete for 8170551 Transformation complete for 2476560 Transformation complete for 19803309 Transformation complete for 18821488 Transformation complete for 8372922 Transformation complete for 9195768 Transformation complete for 15338796 Transformation complete for 12627929 Transformation complete for 6634932 Transformation complete for 18791946 Transformation complete for 17344566 Transformation complete for 9293063 Transformation complete for 1728522 Transformation complete for 10807237 Transformation complete for 12536034 Transformation complete for 7234705 Transformation complete for 8955532 Transformation complete for 2578334 Transformation complete for 11532387 Transformation complete for 3173179 Transformation complete for 3856631 Transformation complete for 2522601 Transformation complete for 8741744 Transformation complete for 17491223 Transformation complete for 18004067 Transformation complete for 12443032 Transformation complete for 17615423 Transformation complete for 8558192 Transformation complete for 11135224 Transformation complete for 8590259 Transformation complete for 2096243 Transformation complete for 6209318 Transformation complete for 3615541 Transformation complete for 8953972 Transformation complete for 3782049 Transformation complete for 9100294 Transformation complete for 19889778 Transformation complete for 15278670 Transformation complete for 17042910 Transformation complete for 11897407 Transformation complete for 9812111 Transformation complete for 15572383 Transformation complete for 7420681 Transformation complete for 12907309 Transformation complete for 16680561 Transformation complete for 17532790 Transformation complete for 7582165 Transformation complete for 16787750 Transformation complete for 3173180 Transformation complete for 8996652 Transformation complete for 84204 Transformation complete for 9862868 Transformation complete for 15515654 Transformation complete for 2024540 Transformation complete for 12090760 Transformation complete for 7516729 Transformation complete for 2802551 Transformation complete for 19346865 Transformation complete for 2334618 Transformation complete for 8864707 Transformation complete for 12589964 Transformation complete for 11263551 Transformation complete for 10520387 Transformation complete for 230316 Transformation complete for 11807648 Transformation complete for 14596845 Transformation complete for 6666578 Transformation complete for 11279304 Transformation complete for 19884587 Transformation complete for 3970039 Transformation complete for 424937 Transformation complete for 8643971 Transformation complete for 2224762 Transformation complete for 2440413 Transformation complete for 12483326 Transformation complete for 1664218 Transformation complete for 18809400 Transformation complete for 17020434 Transformation complete for 3865016 Transformation complete for 3629586 Transformation complete for 150790 Transformation complete for 10579464 Transformation complete for 3496378 Transformation complete for 3076126 Transformation complete for 2894766 Transformation complete for 1549199 Transformation complete for 88336 Transformation complete for 18544179 Transformation complete for 18186898 Transformation complete for 16867021 Transformation complete for 14976857 Transformation complete for 11745184 Transformation complete for 3985451 Transformation complete for 3750012 Transformation complete for 1130930 Transformation complete for 19356307 Transformation complete for 15957009 Transformation complete for 15899738 Transformation complete for 15863244 Transformation complete for 12921865 Transformation complete for 12584269 Transformation complete for 10406016 Transformation complete for 3403780 Transformation complete for 3101906 Transformation complete for 9088814 Transformation complete for 20880751 Transformation complete for 20080419 Transformation complete for 19445921 Transformation complete for 18439803 Transformation complete for 17919553 Transformation complete for 15632880 Transformation complete for 11773892 Transformation complete for 10835440 Transformation complete for 9523805 Transformation complete for 9245658 Transformation complete for 9034419 Transformation complete for 8829025 Transformation complete for 8437969 Transformation complete for 7890216 Transformation complete for 7468724 Transformation complete for 7199841 Transformation complete for 6640832 Transformation complete for 3560096 Transformation complete for 3358181 Transformation complete for 2887062 Transformation complete for 2425813 Transformation complete for 2375138 Transformation complete for 2265898 Transformation complete for 2071257 Transformation complete for 1592014 Transformation complete for 1436384 Transformation complete for 1085609 Transformation complete for 809711 Transformation complete for 20621845 Transformation complete for 19843802 Transformation complete for 19473225 Transformation complete for 19387625 Transformation complete for 19211690 Transformation complete for 18703024 Transformation complete for 18631865 Transformation complete for 18162529 Transformation complete for 17879945 Transformation complete for 17437408 Transformation complete for 17242861 Transformation complete for 16337777 Transformation complete for 15859940 Transformation complete for 15188772 Transformation complete for 15130900 Transformation complete for 12707296 Transformation complete for 12695819 Transformation complete for 12596116 Transformation complete for 11875660 Transformation complete for 11166519 Transformation complete for 8701950 Transformation complete for 7292072 Transformation complete for 4027862 Transformation complete for 2484011 Transformation complete for 2396046 Transformation complete for 2008831 Transformation complete for 1732369 Transformation complete for 234669 Transformation complete for 16740173 Transformation complete for 12464714 Transformation complete for 6203452 Transformation complete for 20528871 Transformation complete for 11672959 Transformation complete for 11928786 Transformation complete for 7977601 Transformation complete for 19300402 Transformation complete for 15325671 Transformation complete for 9746003 Transformation complete for 1992636 Transformation complete for 8387218 Transformation complete for 20698227 Transformation complete for 20477932 Transformation complete for 9495837 Transformation complete for 1468485 Transformation complete for 1009330 Transformation complete for 347884 Transformation complete for 20667451 Transformation complete for 19767176 Transformation complete for 16330293 Transformation complete for 14568327 Transformation complete for 11250767 Transformation complete for 11206082 Transformation complete for 11027905 Transformation complete for 9334596 Transformation complete for 8595686 Transformation complete for 6466532 Transformation complete for 6308277 Transformation complete for 2870085 Transformation complete for 1639466 Transformation complete for 1527456 Transformation complete for 220563 Transformation complete for 137340 Transformation complete for 9931093 Transformation complete for 20633755 Transformation complete for 20067456 Transformation complete for 18464113 Transformation complete for 18308784 Transformation complete for 17931375 Transformation complete for 15649445 Transformation complete for 14698717 Transformation complete for 12617329 Transformation complete for 12523465 Transformation complete for 11961407 Transformation complete for 9401499 Transformation complete for 2273650 Transformation complete for 10683478
代码
文本
[10]
for key, value in data.items():
abstract = value['abstract']
chemical_names = list(value['chemical2id'].keys())
disease_names = list(value['disease2id'].keys())
relations = []
for rel in value['relations']:
chemical_id = rel['chemical']
disease_id = rel['disease']
chemical_name = [name for name, id in value['chemical2id'].items() if id == chemical_id]
disease_name = [name for name, id in value['disease2id'].items() if id == disease_id]
print(f"Transformation complete for {key}")
entry1 = {
"instruction": "You are a biologist AI. I'll give you the abstract of literature. Please identify all the chemical entities in the abstract, and just give me a list of the chemical entities you recognized.",
"input": abstract,
"output": list2str_entity(chemical_name)
}
entry2 = {
"instruction": "You are a biologist AI. I'll give you the abstract of literature. Please identify all the disease entities in the abstract, and just give me a list of the disease entities you recognized.",
"input": abstract,
"output": list2str_entity(disease_name)
}
new_data.append(entry1)
new_data.append(entry2)
Transformation complete for 227508 Transformation complete for 354896 Transformation complete for 435349 Transformation complete for 603022 Transformation complete for 1378968 Transformation complete for 1420741 Transformation complete for 1601297 Transformation complete for 1967484 Transformation complete for 2234245 Transformation complete for 2385256 Transformation complete for 2505783 Transformation complete for 2515254 Transformation complete for 2572625 Transformation complete for 2632720 Transformation complete for 2670794 Transformation complete for 2696505 Transformation complete for 2924746 Transformation complete for 2951327 Transformation complete for 3192036 Transformation complete for 3409645 Transformation complete for 3412544 Transformation complete for 3425586 Transformation complete for 3437726 Transformation complete for 3693336 Transformation complete for 3780846 Transformation complete for 3800626 Transformation complete for 3827439 Transformation complete for 3997294 Transformation complete for 4071154 Transformation complete for 6103707 Transformation complete for 6229975 Transformation complete for 6286738 Transformation complete for 6287825 Transformation complete for 6386793 Transformation complete for 6387529 Transformation complete for 6692345 Transformation complete for 6773726 Transformation complete for 6888657 Transformation complete for 7265370 Transformation complete for 7423039 Transformation complete for 7444978 Transformation complete for 7834920 Transformation complete for 7881871 Transformation complete for 7930386 Transformation complete for 7988234 Transformation complete for 8073369 Transformation complete for 8302922 Transformation complete for 8410052 Transformation complete for 8423889 Transformation complete for 8682684 Transformation complete for 9625142 Transformation complete for 9766615 Transformation complete for 10193204 Transformation complete for 10526274 Transformation complete for 10669626 Transformation complete for 11379838 Transformation complete for 11419773 Transformation complete for 11581460 Transformation complete for 11706060 Transformation complete for 11752354 Transformation complete for 12369736 Transformation complete for 12639165 Transformation complete for 12653683 Transformation complete for 14659530 Transformation complete for 15804801 Transformation complete for 16160878 Transformation complete for 16274958 Transformation complete for 16330766 Transformation complete for 16574712 Transformation complete for 17111419 Transformation complete for 17175308 Transformation complete for 17244258 Transformation complete for 18020536 Transformation complete for 18023325 Transformation complete for 18208574 Transformation complete for 18343374 Transformation complete for 18417364 Transformation complete for 18442015 Transformation complete for 18619688 Transformation complete for 19308880 Transformation complete for 19631624 Transformation complete for 20003049 Transformation complete for 20196116 Transformation complete for 20470218 Transformation complete for 20722491 Transformation complete for 20882060 Transformation complete for 26094 Transformation complete for 322550 Transformation complete for 869641 Transformation complete for 1749407 Transformation complete for 1786266 Transformation complete for 1835291 Transformation complete for 1919871 Transformation complete for 1987816 Transformation complete for 2054792 Transformation complete for 2304736 Transformation complete for 2312209 Transformation complete for 2549018 Transformation complete for 2611118 Transformation complete for 2673163 Transformation complete for 3107448 Transformation complete for 3341566 Transformation complete for 3564823 Transformation complete for 3676049 Transformation complete for 3719553 Transformation complete for 4008111 Transformation complete for 6133211 Transformation complete for 6503301 Transformation complete for 6727060 Transformation complete for 7083920 Transformation complete for 7269015 Transformation complete for 7352670 Transformation complete for 7504976 Transformation complete for 7628595 Transformation complete for 7858459 Transformation complete for 7862923 Transformation complete for 7919560 Transformation complete for 8092427 Transformation complete for 8638206 Transformation complete for 8669433 Transformation complete for 8690168 Transformation complete for 8739323 Transformation complete for 8752018 Transformation complete for 9390208 Transformation complete for 9406968 Transformation complete for 9587734 Transformation complete for 9698967 Transformation complete for 9855119 Transformation complete for 9869257 Transformation complete for 10342929 Transformation complete for 10457883 Transformation complete for 10739826 Transformation complete for 10791295 Transformation complete for 11147747 Transformation complete for 11198499 Transformation complete for 11391224 Transformation complete for 11426838 Transformation complete for 11569530 Transformation complete for 11587867 Transformation complete for 11679859 Transformation complete for 12041669 Transformation complete for 12198388 Transformation complete for 12574103 Transformation complete for 12615818 Transformation complete for 12789195 Transformation complete for 12820454 Transformation complete for 14657095 Transformation complete for 14765563 Transformation complete for 15036754 Transformation complete for 15145918 Transformation complete for 15233872 Transformation complete for 15458908 Transformation complete for 15737522 Transformation complete for 16005948 Transformation complete for 16167916 Transformation complete for 16403073 Transformation complete for 16755009 Transformation complete for 16904497 Transformation complete for 16938416 Transformation complete for 17147461 Transformation complete for 17241784 Transformation complete for 17261653 Transformation complete for 17343925 Transformation complete for 17400887 Transformation complete for 17612891 Transformation complete for 18081909 Transformation complete for 18083142 Transformation complete for 18182964 Transformation complete for 18217897 Transformation complete for 18996674 Transformation complete for 19058010 Transformation complete for 19581773 Transformation complete for 19759529 Transformation complete for 19957053 Transformation complete for 20619828 Transformation complete for 891050 Transformation complete for 983936 Transformation complete for 1428568 Transformation complete for 1720453 Transformation complete for 1833784 Transformation complete for 1867351 Transformation complete for 2070391 Transformation complete for 2266990 Transformation complete for 2348231 Transformation complete for 2355241 Transformation complete for 2445283 Transformation complete for 2950248 Transformation complete for 3015567 Transformation complete for 3031535 Transformation complete for 3125768 Transformation complete for 3187073 Transformation complete for 3371379 Transformation complete for 3503576 Transformation complete for 3560095 Transformation complete for 3714122 Transformation complete for 3828020 Transformation complete for 4812392 Transformation complete for 6518066 Transformation complete for 6534871 Transformation complete for 6538499 Transformation complete for 6747681 Transformation complete for 6861444 Transformation complete for 7053303 Transformation complete for 7088431 Transformation complete for 7248170 Transformation complete for 7453952 Transformation complete for 7802851 Transformation complete for 8319760 Transformation complete for 8386779 Transformation complete for 8421099 Transformation complete for 8649546 Transformation complete for 8919272 Transformation complete for 9199746 Transformation complete for 9249847 Transformation complete for 9284778 Transformation complete for 9522143 Transformation complete for 9522152 Transformation complete for 9672936 Transformation complete for 9721172 Transformation complete for 9727773 Transformation complete for 9754849 Transformation complete for 10365197 Transformation complete for 10704919 Transformation complete for 10706004 Transformation complete for 10721819 Transformation complete for 10737864 Transformation complete for 10743446 Transformation complete for 11007689 Transformation complete for 11256525 Transformation complete for 11334364 Transformation complete for 11642480 Transformation complete for 12063090 Transformation complete for 12084448 Transformation complete for 12684739 Transformation complete for 12716030 Transformation complete for 12757899 Transformation complete for 12905102 Transformation complete for 14513889 Transformation complete for 15009014 Transformation complete for 15120741 Transformation complete for 15275829 Transformation complete for 15602202 Transformation complete for 16181582 Transformation complete for 16298782 Transformation complete for 16428221 Transformation complete for 16858720 Transformation complete for 17466854 Transformation complete for 17562951 Transformation complete for 17600377 Transformation complete for 17639754 Transformation complete for 17854040 Transformation complete for 18221780 Transformation complete for 18261172 Transformation complete for 18329269 Transformation complete for 18410508 Transformation complete for 18503483 Transformation complete for 18560792 Transformation complete for 18726058 Transformation complete for 18754075 Transformation complete for 18768591 Transformation complete for 19299179 Transformation complete for 19356053 Transformation complete for 19729346 Transformation complete for 20080983 Transformation complete for 20520283 Transformation complete for 20588063 Transformation complete for 19820426 Transformation complete for 1616457 Transformation complete for 567256 Transformation complete for 17439425 Transformation complete for 9351491 Transformation complete for 18752389 Transformation complete for 20098969 Transformation complete for 9653867 Transformation complete for 17608141 Transformation complete for 2559236 Transformation complete for 21029050 Transformation complete for 10901305 Transformation complete for 19037603 Transformation complete for 20084309 Transformation complete for 8231633 Transformation complete for 10091617 Transformation complete for 19269743 Transformation complete for 3070035 Transformation complete for 1147734 Transformation complete for 12202650 Transformation complete for 18951540 Transformation complete for 19657887 Transformation complete for 9660111 Transformation complete for 11431197 Transformation complete for 7449470 Transformation complete for 8170551 Transformation complete for 2476560 Transformation complete for 19803309 Transformation complete for 18821488 Transformation complete for 8372922 Transformation complete for 9195768 Transformation complete for 15338796 Transformation complete for 12627929 Transformation complete for 6634932 Transformation complete for 18791946 Transformation complete for 17344566 Transformation complete for 9293063 Transformation complete for 1728522 Transformation complete for 10807237 Transformation complete for 12536034 Transformation complete for 7234705 Transformation complete for 8955532 Transformation complete for 2578334 Transformation complete for 11532387 Transformation complete for 3173179 Transformation complete for 3856631 Transformation complete for 2522601 Transformation complete for 8741744 Transformation complete for 17491223 Transformation complete for 18004067 Transformation complete for 12443032 Transformation complete for 17615423 Transformation complete for 8558192 Transformation complete for 11135224 Transformation complete for 8590259 Transformation complete for 2096243 Transformation complete for 6209318 Transformation complete for 3615541 Transformation complete for 8953972 Transformation complete for 3782049 Transformation complete for 9100294 Transformation complete for 19889778 Transformation complete for 15278670 Transformation complete for 17042910 Transformation complete for 11897407 Transformation complete for 9812111 Transformation complete for 15572383 Transformation complete for 7420681 Transformation complete for 12907309 Transformation complete for 16680561 Transformation complete for 17532790 Transformation complete for 7582165 Transformation complete for 16787750 Transformation complete for 3173180 Transformation complete for 8996652 Transformation complete for 84204 Transformation complete for 9862868 Transformation complete for 15515654 Transformation complete for 2024540 Transformation complete for 12090760 Transformation complete for 7516729 Transformation complete for 2802551 Transformation complete for 19346865 Transformation complete for 2334618 Transformation complete for 8864707 Transformation complete for 12589964 Transformation complete for 11263551 Transformation complete for 10520387 Transformation complete for 230316 Transformation complete for 11807648 Transformation complete for 14596845 Transformation complete for 6666578 Transformation complete for 11279304 Transformation complete for 19884587 Transformation complete for 3970039 Transformation complete for 424937 Transformation complete for 8643971 Transformation complete for 2224762 Transformation complete for 2440413 Transformation complete for 12483326 Transformation complete for 1664218 Transformation complete for 18809400 Transformation complete for 17020434 Transformation complete for 3865016 Transformation complete for 3629586 Transformation complete for 150790 Transformation complete for 10579464 Transformation complete for 3496378 Transformation complete for 3076126 Transformation complete for 2894766 Transformation complete for 1549199 Transformation complete for 88336 Transformation complete for 18544179 Transformation complete for 18186898 Transformation complete for 16867021 Transformation complete for 14976857 Transformation complete for 11745184 Transformation complete for 3985451 Transformation complete for 3750012 Transformation complete for 1130930 Transformation complete for 19356307 Transformation complete for 15957009 Transformation complete for 15899738 Transformation complete for 15863244 Transformation complete for 12921865 Transformation complete for 12584269 Transformation complete for 10406016 Transformation complete for 3403780 Transformation complete for 3101906 Transformation complete for 9088814 Transformation complete for 20880751 Transformation complete for 20080419 Transformation complete for 19445921 Transformation complete for 18439803 Transformation complete for 17919553 Transformation complete for 15632880 Transformation complete for 11773892 Transformation complete for 10835440 Transformation complete for 9523805 Transformation complete for 9245658 Transformation complete for 9034419 Transformation complete for 8829025 Transformation complete for 8437969 Transformation complete for 7890216 Transformation complete for 7468724 Transformation complete for 7199841 Transformation complete for 6640832 Transformation complete for 3560096 Transformation complete for 3358181 Transformation complete for 2887062 Transformation complete for 2425813 Transformation complete for 2375138 Transformation complete for 2265898 Transformation complete for 2071257 Transformation complete for 1592014 Transformation complete for 1436384 Transformation complete for 1085609 Transformation complete for 809711 Transformation complete for 20621845 Transformation complete for 19843802 Transformation complete for 19473225 Transformation complete for 19387625 Transformation complete for 19211690 Transformation complete for 18703024 Transformation complete for 18631865 Transformation complete for 18162529 Transformation complete for 17879945 Transformation complete for 17437408 Transformation complete for 17242861 Transformation complete for 16337777 Transformation complete for 15859940 Transformation complete for 15188772 Transformation complete for 15130900 Transformation complete for 12707296 Transformation complete for 12695819 Transformation complete for 12596116 Transformation complete for 11875660 Transformation complete for 11166519 Transformation complete for 8701950 Transformation complete for 7292072 Transformation complete for 4027862 Transformation complete for 2484011 Transformation complete for 2396046 Transformation complete for 2008831 Transformation complete for 1732369 Transformation complete for 234669 Transformation complete for 16740173 Transformation complete for 12464714 Transformation complete for 6203452 Transformation complete for 20528871 Transformation complete for 11672959 Transformation complete for 11928786 Transformation complete for 7977601 Transformation complete for 19300402 Transformation complete for 15325671 Transformation complete for 9746003 Transformation complete for 1992636 Transformation complete for 8387218 Transformation complete for 20698227 Transformation complete for 20477932 Transformation complete for 9495837 Transformation complete for 1468485 Transformation complete for 1009330 Transformation complete for 347884 Transformation complete for 20667451 Transformation complete for 19767176 Transformation complete for 16330293 Transformation complete for 14568327 Transformation complete for 11250767 Transformation complete for 11206082 Transformation complete for 11027905 Transformation complete for 9334596 Transformation complete for 8595686 Transformation complete for 6466532 Transformation complete for 6308277 Transformation complete for 2870085 Transformation complete for 1639466 Transformation complete for 1527456 Transformation complete for 220563 Transformation complete for 137340 Transformation complete for 9931093 Transformation complete for 20633755 Transformation complete for 20067456 Transformation complete for 18464113 Transformation complete for 18308784 Transformation complete for 17931375 Transformation complete for 15649445 Transformation complete for 14698717 Transformation complete for 12617329 Transformation complete for 12523465 Transformation complete for 11961407 Transformation complete for 9401499 Transformation complete for 2273650 Transformation complete for 10683478
代码
文本
任务三
代码
文本
[11]
with open(DATA_PATH + 'task3/train.json', 'r') as file:
data = json.load(file)
instruction_text = "You are a medicinal chemist. Now you need to identify all the drug-drug interactions from the text I provide to you, and please only write down all the drug-drug interactions in the format of (drug, interaction, drug). If there are more than one triple, please write in this form: '(drug, interaction, drug),(drug, interaction, drug),......'"
for key, entry in data.items():
input_text = entry['abstract'] # Extract 'abstract' for 'input'
triples_text = ', '.join([f"({triple['drug']}, {triple['interaction']}, {triple['target']})" for triple in entry['triples']])
new_entry = {
"instruction": instruction_text,
"input": input_text,
"response": triples_text
}
new_data.append(new_entry)
代码
文本
写入data.json
代码
文本
把所有处理过后的数据全部写入data.json
代码
文本
[12]
with open('data.json', 'w') as file:
json.dump(new_data, file, indent=4)
代码
文本
拆分训练集和验证集
以上，你已经得到了用于微调的data.json啦！现在，请你将它按照你自己理解的认为对的拆分方式，拆分成data_train.json和data.valid.json，作为下一个notebook的输出吧！
下一个notebook看这里：https://nb.bohrium.dp.tech/detail/35979425971
代码
文本
点个赞吧
本文被以下合集收录
LLM
鹏举瞰雨
更新于 2024-04-03
1 篇0 人关注
LLMs
bohr690dff
更新于 2024-03-17
2 篇0 人关注
推荐阅读
公开
Unifold Batch Inferencecsg
发布于 2023-09-24
2 赞5 转存文件
公开
Notebook 可视化-DPMD 训练损失图和学习率变化曲线
Hui_Zhou
发布于 2023-10-17
2 赞2 转存文件