python – 使用sqlalchemy将CSV导入数据库

前端之家收集整理的这篇文章主要介绍了python – 使用sqlalchemy将CSV导入数据库前端之家小编觉得挺不错的,现在分享给大家,也给大家做个参考。
我使用 this示例将csv文件上传sqlite数据库

这是我的代码

  1. from numpy import genfromtxt
  2. from time import time
  3. from datetime import datetime
  4. from sqlalchemy import Column,Integer,Float,Date,String,VARCHAR
  5. from sqlalchemy.ext.declarative import declarative_base
  6. from sqlalchemy import create_engine
  7. from sqlalchemy.orm import sessionmaker
  8.  
  9. def Load_Data(file_name):
  10. data = genfromtxt(file_name,delimiter=',')# skiprows=1,converters={0: lambda s: str(s)})
  11. return data.tolist()
  12.  
  13. Base = declarative_base()
  14.  
  15. class cdb1(Base):
  16. #Tell sqlAlchemy what the table name is and if there's any table-specific arguments it should know about
  17. __tablename__ = 'cdb1'
  18. __table_args__ = {'sqlite_autoincrement': True}
  19. #tell sqlAlchemy the name of column and its attributes:
  20. id = Column(Integer,primary_key=True,nullable=False)
  21. name = Column(VARCHAR(40))
  22. shack = Column(VARCHAR)
  23. db = Column(Integer)
  24. payments = Column(Integer)
  25. status = Column(VARCHAR)
  26.  
  27.  
  28. if __name__ == "__main__":
  29. t = time()
  30. print 'creating database'
  31.  
  32. #Create the database
  33. engine = create_engine('sqlite:///cdb.db')
  34. Base.Metadata.create_all(engine)
  35.  
  36. #Create the session
  37. session = sessionmaker()
  38. session.configure(bind=engine)
  39. s = session()
  40.  
  41. try:
  42. file_name = 'client_db.csv'
  43. data = Load_Data(file_name)
  44.  
  45. for i in data:
  46. record = cdb1(**{
  47. 'name' : i[0],'shack' : i[1],'db' : i[2],'payments' : i[3],'status' : i[4]
  48. })
  49. s.add(record) #Add all the records
  50.  
  51. s.commit() #Attempt to commit all the records
  52. except:
  53. s.rollback() #Rollback the changes on error
  54. print 'error in reading'
  55. finally:
  56. s.close() #Close the connection
  57. print "Time elapsed: " + str(time() - t) + " s." #0.091s

这是csv文件的前几行:

  1. Name,Shack,DB,Payments,Status
  2. Loyiso Dwala,I156,13542,37,LightsOnly ON
  3. Attwell Fayo,I157,13077,32,LightsON
  4. David Mbhele,G25,13155,33,LightsON

数据库创建正常,但只有部分数据被捕获到属性中:’payments’和’db’列被正确填充,但其他所有数据都是NULL.

更新的正确代码(使用pandas数据帧):

  1. from numpy import genfromtxt
  2. from time import time
  3. from datetime import datetime
  4. from sqlalchemy import Column,VARCHAR
  5. from sqlalchemy.ext.declarative import declarative_base
  6. from sqlalchemy import create_engine
  7. from sqlalchemy.orm import sessionmaker
  8. import csv
  9. import pandas as pd
  10.  
  11.  
  12. #def Load_Data(file_name):
  13. #data = csv.reader(file_name,converters={0: lambda s: str(s)})
  14. #return data.tolist()
  15.  
  16. Base = declarative_base()
  17.  
  18. class cdb1(Base):
  19. #Tell sqlAlchemy what the table name is and if there's any table-specific arguments it should know about
  20. __tablename__ = 'cdb1'
  21. __table_args__ = {'sqlite_autoincrement': True}
  22. #tell sqlAlchemy the name of column and its attributes:
  23. id = Column(Integer,nullable=False)
  24. Name = Column(VARCHAR(40))
  25. Shack = Column(VARCHAR)
  26. DB = Column(Integer)
  27. Payments = Column(Integer)
  28. Status = Column(VARCHAR)
  29.  
  30. engine = create_engine('sqlite:///cdb.db')
  31. Base.Metadata.create_all(engine)
  32. file_name = 'client_db.csv'
  33. df = pd.read_csv(file_name)
  34. df.to_sql(con=engine,index_label='id',name=cdb1.__tablename__,if_exists='replace')

解决方法

你熟悉Pandas Dataframe吗?

真的很简单(和调试)

pandas.read_csv(file_name)

  1. In [5]: pandas.read_csv('/tmp/csvt.csv')
  2. Out[5]:
  3. Name Shack DB Payments Status
  4. 0 Loyiso Dwala I156 13542 37 LightsOnly ON
  5. 1 Attwell Fayo I157 13077 32 LightsON
  6. 2 David Mbhele G25 13155 33 LightsON

要将DataFrames数据插入表中,只需使用pandas.DataFrame.to_sql即可

所以你的主要代码最终会看起来像这样:

  1. engine = create_engine('sqlite:///cdb.db')
  2. Base.Metadata.create_all(engine)
  3.  
  4. file_name = 'client_db.csv'
  5. df = pandas.read_csv(file_name)
  6. df.to_sql(con=engine,if_exists='replace')

您应该在我添加的文档链接中进一步阅读,并将函数Parameters设置为适合您的目的(特别是 – if_exists,index,index_label,dtype)

猜你在找的Python相关文章