Issue with memory pointers in class mapping for simplified SELECT queries

I'm working on improving the SELECT functionality in my application by mapping a class and storing relevant information into a record. Here is an example of my class properties:

[Field('ID'), PK, AutoInc]
property ID: Int64 read FID write FID;
[Field('NOTA_ORIGINAL')]
property NotaOriginal: string read FNotaOriginal write FNotaOriginal;
[Field('NOTA_TROCA')]
property NotaTroca: string read FNotaTroca write FNotaTroca;

Part of my mapping function looks like this:

for lProperty in lType.GetProperties do
begin
  if (lProperty.HasField<Field>) then
  begin
    lMappedField.Field := lProperty.SearchAttribute<Field>.FieldName;
    lMappedField.PK := False;

    if (lProperty.HasAttribute<PK>) then
    begin
      lMappedField.PK := True;
      FPKKey := lProperty.SearchAttribute<Field>.FieldName;
      FPropertyPK := lProperty;
    end;

    lMappedField.AutoInc := (lProperty.HasAttribute<AutoInc>);

    if lProperty is TRttiInstanceProperty then
    begin
      PropInstance := TRttiInstanceProperty(lProperty);
      MemAddress := nil;

      if PropInstance.IsReadable then
      begin
        // Check if lProperty is directly tied to a field
        if (IntPtr(PropInstance.PropInfo^.GetProc) and PROPSLOT_MASK) = PROPSLOT_FIELD then
        begin
          // Obtain the memory address of the directly associated field
          MemAddress := PByte(FObjeto) + (IntPtr(PropInstance.PropInfo^.GetProc) and (not PROPSLOT_MASK));
          lMappedField.Mem := MemAddress;
        end;
      end;
    end;

    FMapping.Add(lProperty, lMappedField);
  end;
end;

Here’s the record structure I'm using:

TMappedField = record
  Field: string;
  PK: Boolean;
  AutoInc: Boolean;
  Mem: Pointer;
end;

The mapping is stored in a variable: FMapping: TDictionary<TRttiProperty, TMappedField>. I also have other variables for quick access to key information:

FSQLTable: string; // class attribute
FPKKey: string; 
FPropertyPK: TRttiProperty;

This mapping system works well and saves time for various classes. However, I have a concern regarding the Mem pointer that I save, which I use to simplify the SELECT queries. Here is an example:

function TMCRTTILibORM.Load: TMCRTTILibORM;
begin
  FSelectText.Clear;
  FSelectText.Add(' SELECT FIRST 1 ' + FPKKey + ' FROM ' + FSQLTable + ' ');
  Result := Self;
end;

function TMCRTTILibORM.Where(const pProperty: Pointer; const pValue: Variant): TMCRTTILibORM;
var
  lPropertyField: TPair<TRttiProperty, TMappedField>;
begin
  for lPropertyField in FMapping do
  begin
    if lPropertyField.Value.Mem = pProperty then
    begin
      // Add WHERE or AND to SQL query
      if FSelectText.Count > 1 then
        FSelectText.Add(' AND ' + lPropertyField.Value.Field + ' = ')
      else
        FSelectText.Add(' WHERE ' + lPropertyField.Value.Field + ' = ');

      // Handle different types (Integer, Float, String, Enumeration, etc.)
      // Example for tkInteger:
      FSelectText.Add(VarToStr(pValue)); 

      Result := Self;
      Exit;
    end;
  end;
end;

function TMCRTTILibORM.Open: Boolean;
var
  lQuery: TCustomQuery;
begin
  lQuery := TCustomQuery.Create(nil);
  try
    lQuery.SQL := FSelectText;
    lQuery.Open;

    // Update object with primary key value
    if lQuery.RecordCount > 0 then
    begin
      FPropertyPK.SetValue(FObjeto, lQuery.FieldByName(FPKKey).AsInteger); // Example for tkInteger
      Result := LoadObject; // Populate the object
    end;
  finally
    lQuery.Free;
  end;
end;

With this, I can simply call:

Result := ItemObject.Mapping.Load
  .Where(@ItemObject.ID, pID)
  .Open;

Concern: A senior developer mentioned that the Mem pointer stored in my record could be invalidated by Windows if memory is managed or relocated, causing the comparison in the Where function to fail.

My current approach maps the object only once and retrieves its attributes on-demand. All my tests so far have been successful, but if this memory management issue is real, I may not be able to rely on this approach.

Question: Is the concern about the memory pointer (Mem) valid in this scenario? Could Windows memory management indeed alter the pointer, and if so, what would be a better way to maintain this mapping for select queries?