Updated Approach for a Less-Hassle ORM Attributes Management

4 minute read

Published:

In the previous article I wrote about how I refactored the attributes management approach for Object Relational Mapper (ORM) use case. You can find the article here.

After discussing a lot with a friend of mine, the refactoring approach was changed a bit. The point is, how to enable the developers and the users handle all the relevant attributes in a minimum number of modules.

In the previous article, the general approach goes like the following:

  • All attributes with specific properties (primary key, unique, autoincrement, etc.) are specified in the ORM module
  • All attributes without specific properties are specified in the Repo modules. These attributes are then combined with all the specific attributes

As a refresher, here are an example of the ORM and Repo module.

File: TableORM.py

from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()


class TableORM(Base):
	__tablename__ = my_table
	__table__ = Table(__tablename__, Base.metadata, Column(field_a, String, primary_key=True))

File: TableRepository.py

from TableConfig import get_all_columns

class TableRepository(object):
	__metaclass__ = abc.ABCMeta

	def __init__(self):
		self._columns = get_all_columns()
    
                preserved_columns = ['field_a']
		orm_columns = []
		for column in self._columns:
			if column not in preserved_columns:
				column_type = ORM_SCHEMA_MAPPING[DATA_SCHEMA[column]]
				orm_columns.append(Column(column, column_type))

		TableORM.__table__ = Table(my_table, Base.metadata, *orm_columns, extend_existing=True)

	@abc.abstractmethod
	def upsert(self, df: DataFrame):
		pass

From the above code snippets, we can see that the ORM module has one specific attribute, that is field_a with property of primary key. Meanwhile, the Repo module stores all the non-specific attributes. After all the non-specific attributes are retrieved, the Table object is updated by specifying an extend_existing argument. This argument enables us to append the additional columns with the existing ones.

Since the number of specific columns is not that many, the hassle of specifying all the specific columns in the ORM module and preserved_columns list in the Repo module should be quiet low. However, the problem is, we still need to maintain two modules if there’s any change on the attributes. FYI, the columns specified in the ORM module must be the same with the ones specified in preserved_columns list in the Repo module.

Without further ado, here’s the updated approach.

UPDATED APPROACH

File: TableConfig.py

def get_all_columns():
	return [
		field_a, field_b, field_c, field_d, 
		
		
		field_final
	]

File: StorageUtil.py

class MyTableUtil(object):
  @staticmethod
  def get_orm_columns(preserved_columns, repo_columns):
      orm_columns = []
		  
      for column in repo_columns:
          if column not in preserved_columns:
              column_type = ORM_SCHEMA_MAPPING[DATA_SCHEMA[column]]
              orm_columns.append(Column(column, column_type))
      
      return orm_columns

File: TableORM.py

from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base

from TableConfig import get_all_columns
from StorageUtil import MyTableUtil

Base = declarative_base()


class TableORM(Base):

  __tablename__ = my_table
  
  @staticmethod
  def _get_columns():
      # stores all the specific columns
      columns = [
          Column('field_a', String, primary_key=True)
      ]
      
      return columns + MyTableUtil.get_orm_columns(preserved_columns, repo_columns)
      
  @staticmethod
  def _construct_table(columns):
      return Table(__tablename__, Base.metadata, *columns)
  
  preserved_columns = ['field_a']
  repo_columns = get_all_columns()
  
  columns = _get_columns.__func__()
  __table__ = _construct_table.__func__(columns)

With this new approach, if there’s any change on attributes specified in TableConfig.py module, one only needs to list all the specific attributes in preserved_columns and columns (in _get_columns method) in TableORM.py.

If the previous approach requires us to work on two different modules - ORM and Repo, then this updated approach enables us to do all the process in a single file only.

However, we still need to get the list of columns for TableRepository.py module. Here’s the code snippet before the updated approach is applied for a refresher.

File: TableRepository.py

from TableConfig import get_all_columns

class TableRepository(object):
	__metaclass__ = abc.ABCMeta

	def __init__(self):
                # we still need to retrieve the list of columns from TableConfig
		self._columns = get_all_columns()

		orm_columns = []
		for column in self._columns:
			if column != field_a:
				column_type = ORM_SCHEMA_MAPPING[DATA_SCHEMA[column]]
				orm_columns.append(Column(column, column_type))

		TableORM.__table__ = Table(my_table, Base.metadata, *orm_columns, extend_existing=True)

Basically, using the above code snippet would not be a big problem as changes on attributes are made on TableConfig.py module. We just need to call get_all_columns method to retrieve them. However, how if we need additional attributes. In other words, we would like to make self._columns to store the value of get_all_columns() + additional_columns.

The above problem requires us to make a modification on TableRepository.py as well as TableORM.py. In TableORM.py, we’ll need to assign repo_columns attribute to get_all_columns() + additional_columns. Implicitly, we’ll need to specify all the additional columns in both TransactionORM.py and TransactionRepository.py modules.

To address this problem, I think we can utilise the repo_columns attribute from TransactionORM.py. Long story short, it should be like the following.

File: TableRepository.py

from TableConfig import get_all_columns
from TableORM import TableORM


class TableRepository(object):
	__metaclass__ = abc.ABCMeta

	def __init__(self):
		self._columns = TableORM.repo_columns

VOILA!

I guess this is the end of this article. Hope it helps you.

Thank you for reading.