.tommasop:

I’m using a BulkCreate action but it seems not to upsert correctly:

.tommasop:

I also tried disabling Carbonite with same result.

.tommasop:

Sorry for using screenshots but it was the easiest way to show the problem.

zachdaniel:

When you say not working what do you mean?

zachdaniel:

It doesn’t return results by default, that’s an option you have to pass

frankdugan3:

I’m trying out the bulk creates with upserts and returned results for my staging seeds file, working well so far. 🙂

frankdugan3:

And I have to say, it’s a fantastic API for that purpose. 🚀

.tommasop:

I’m passing return_records?: true and still it’s not returning the upserted record and the product record is not updated

frankdugan3:

Do you have stop_on_error?: true ? Because otherwise it might not display an error that’s causing a problem (that’s been my experience).

zachdaniel:

Oh interesting…that sounds like a bug

zachdaniel:

<@433654314175692800> do you have a reproduction for the errors not appearing that you mentioned?

frankdugan3:

I don’t remember the specific circumstance… I think it was on initial create when giving some bad fields as params… If I run into it again I’ll create an issue now that I know it should error.

zachdaniel:

it should return the errors at least 🙂

frankdugan3:

Yeah, I may have had both those options off when that happened. lol

.tommasop:

<@433654314175692800> you are right with stop_on_error?: true I have the error still there is something not working as I expect (but maybe as it should 😆 )

.tommasop:

I have a :create action (also used for upsert) for a Product that needs a data_in and a category_id relationships.

.tommasop:

there is an API call used only to update product quantities that returns a list of %{sku_code: "MAGP0.6.2", quantity: 14.0}

.tommasop:

I have a :unique_sku_code identity in product

.tommasop:

I add data_in_id and category_id making data become a list of [%{category_id: "b77fa8c6-330e-422f-8ee5-84277096821b", data_in_id: "221408ba-ff99-46ce-a01d-bccacc3579c7", quantity: 14.0, sku_code: "MAGP0.6.2"}]

.tommasop:

I would expect this bulk_create to only update the product quantity but it gives me an error:

MmsBiztalk.bulk_create!(
        [%{category_id: "b77fa8c6-330e-422f-8ee5-84277096821b", data_in_id: "221408ba-ff99-46ce-a01d-bccacc3579c7", quantity: 14.0, sku_code: "MAGP0.6.2"}],
        MmsBiztalk.Product,
        :create,
        upsert?: true,
        upsert_identity: :unique_sku_code,
        # for bulk actions upsert_fields must be specified
        upsert_fields: [:quantity],
        stop_on_error?: true,
        return_records?: true
      )

to update only the product quantity instead it gives me an error where it is trying to update all the fields (setting nil to some needed ones).

.tommasop:

[%Ash.Error.Unknown{errors: [%Ash.Error.Unknown.UnknownError{error: "** (Postgrex.Error) ERROR 23502 (not_null_violation) null value in column \"description_en\" of r
elation \"products\" violates not-null constraint\n\n    table: products\n    column: description_en\n\nFailing row contains (05ef01a1-1e12-4295-972f-656c431f6578, MAGP0.6.2, null, nul
l, null, null, null, null, null, null, null, null, null, null, null, null, null, 2023-05-25 09:06:32.841523, 2023-05-25 09:06:32.841523, 14, not_synced, null).", field: nil, changeset:
 nil, query: nil, error_context: [], vars: [], path: [:create_or_update_resources], stacktrace: #Stacktrace<>, class: :unknown}], stacktraces?: true, changeset: nil, query: nil, error_
context: [], vars: [], path: [], stacktrace: #Stacktrace<>, class: :unknown}]

.tommasop:

Do I need to pass also the Product id to make the upsert work even if I have the :unique_sku_code identity?

.tommasop:

Or am I using the upsert in an unproper way?

zachdaniel:

okay so I need to make sure that error shows up even if stop_on_error? is false

.tommasop:

apart from error rising am I using the upsert correctly and if so why isn’t the poduct quantity updated?

frankdugan3:

I don’t see :unique_sku_code in your create data: How would it be able to compare it against the existing row if that’s the identity field? <:thinkies:915154230078222336>

frankdugan3:

Oh, is that the name your identity for :sku_code ? I wonder if it wants the field name instead.

.tommasop:

this is the :unique_sku_code identity

.tommasop:

identities do
    identity :unique_sku_code, :sku_code, eager_check_with: MmsBiztalk
  end

frankdugan3:

I’m not sure because I generally name my single-key identities the same as the field.

.tommasop:

ok I can try that easily

.tommasop:

no I have the same error giving :sku_code as the upsert_identity

zachdaniel:

okay so I made some small fixes, commented about in the announcement for bulk creates

zachdaniel:

specifically we were actually not honoring the return_errors? option, and it was supposed to default to false

zachdaniel:

which is technically a minor breaking change but this is a new-ish feature and I’d rather not go through the whole rigamarole of the breaking change process for it.

zachdaniel:

lemme look over this again and see if I can spot the issue

zachdaniel:

also <@1042733068168986684> are you on the latest ash and ash_postgres?

.tommasop:

I am on main

.tommasop:

in both

zachdaniel:

👌

zachdaniel:

Actually I don’t think I’ve seen the bulk insert generated sql statement yet

zachdaniel:

Is that in the logs? Did I just miss it?

zachdaniel:

If we can see why that’s wrong the fix should be pretty clear

.tommasop:

I’m using a list with a single product to test the behavior and with a LOG_LEVEL=debug this is what I see in iex

MmsBiztalk.bulk_create!(input.resources_attributes, input.resource, :create, upsert?: true, upsert_identity: :unique_sku_code, upsert_fields: [:sku_code, :quantity], return_records?: true)
** (throw) {DBConnection, #Reference<0.1826028390.1079508994.123791>, %Ash.Error.Unknown.UnknownError{error: "** (Postgrex.Error) ERROR 23502 (not_null_violation) null value in column \"description_en\" of relation \"products\" violates not-null constraint\n\n    table: products\n    column: description_en\n\nFailing row contains (0b584a6a-5cc9-4f45-9a49-b360b67a512b, MAGP0.6.2, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, 2023-05-25 14:21:28.74044, 2023-05-25 14:21:28.74044, 14, not_synced, null).", field: nil, changeset: nil, query: nil, error_context: [], vars: [], path: [], stacktrace: #Stacktrace<>, class: :unknown}}
    (db_connection 2.5.0) lib/db_connection.ex:996: DBConnection.rollback/2
    (ash 2.9.11) lib/ash/data_layer/data_layer.ex:288: Ash.DataLayer.transaction/4
    (ash 2.9.11) lib/ash/actions/create/bulk.ex:191: anonymous fn/7 in Ash.Actions.Create.Bulk.do_run/5
    (elixir 1.14.1) lib/stream.ex:612: anonymous fn/4 in Stream.map/2
    (elixir 1.14.1) lib/enum.ex:4751: Enumerable.List.reduce/3
    (elixir 1.14.1) lib/stream.ex:1026: Stream.do_transform_inner_list/7
    (elixir 1.14.1) lib/stream.ex:1811: Enumerable.Stream.do_each/4
    (elixir 1.14.1) lib/enum.ex:4307: Enum.reduce/3
    (ash 2.9.11) lib/ash/actions/create/bulk.ex:405: Ash.Actions.Create.Bulk.do_run/5
    (ash 2.9.11) lib/ash/actions/create/bulk.ex:69: Ash.Actions.Create.Bulk.run/5
    (ash 2.9.11) lib/ash/api/api.ex:1878: Ash.Api.bulk_create!/5
    /home/tommasop/code/work/magic/mms_biztalk/lib/mms_biztalk/flows/steps/upsert_resources.ex:1: (file)

zachdaniel:

🤔

zachdaniel:

THat is pretty strange

zachdaniel:

Usually there is a log for the actual query

zachdaniel:

Try doing this

zachdaniel:

try do
  MmsBiztalk.bulk_create!(input.resources_attributes, input.resource, :create, upsert?: true, upsert_identity: :unique_sku_code, upsert_fields: [:sku_code, :quantity], return_records?: true)
catch
  thing ->
    thing
end

zachdaniel:

I’m wondering if the thrown error is causing a crash or somethign

zachdaniel:

and its not getting to log the initial query attempt

zachdaniel:

perhaps you can get query logs from the database somehow?

zachdaniel:

We have it working in ash_postgres tests, producing queries like:

INSERT INTO "posts" ("created_at","decimal","id","price","title","type","uniq_one","uniq_two","updated_at") VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9),($10,$11,$12,$13,$14,$15,$16,$17,$18) ON CONFLICT (uniq_one, uniq_two) WHERE (type = 'sponsored') DO UPDATE SET "price" = EXCLUDED."price" RETURNING "author_id","organization_id","updated_at","created_at","uniq_custom_two","uniq_custom_one","uniq_two","uniq_one","point","status_enum","status_enum","status","decimal","price","type","category","public","score","title","id" [~U[2023-05-25 14:45:01.312299Z], Decimal.new("0"), "122ba2db-c858-4347-91e7-e39bd564ff05", 1000, "something", :sponsored, "one", "two", ~U[2023-05-25 14:45:01.312299Z], ~U[2023-05-25 14:45:01.312250Z], Decimal.new("0"), "4e694979-ee64-4093-9032-63feccaa932d", 20000, "else", :sponsored, "three", "four", ~U[2023-05-25 14:45:01.312250Z]]

zachdaniel:

and to be 100% sure you’ve updated to latest ash and ash_postgres main with mix deps.update ash ash_postgres ? I ask that question a lot, but its always worth asking 😆 Sorry if its repetetive.

zachdaniel:

https://giphy.com/gifs/the-it-crowd-DUtVdGeIU8lmo

.tommasop:

double checked it and I have the latest ash and ash_postgres

frankdugan3:

Should I be able to upsert with :id as the identity?

frankdugan3:

uuid_primary_key :id, writable?: true

zachdaniel:

IIRC that is the default behavior

zachdaniel:

<@1042733068168986684> can you see if it works if you use the id to upsert (by not providing an upsert_identity ?)

.tommasop:

of course

.tommasop:

it gives me the same error

zachdaniel:

.tommasop:

pry(1)> MmsBiztalk.bulk_create!(input.resources_attributes, input.resource, :create, upsert?: true, upsert_fields: [:sku_code, :quantity], return_records?: true)                             ** (throw) {DBConnection, #Reference<0.1164303867.3233546242.136685>, %Ash.Error.Unknown.UnknownError{error: "** (Postgrex.Error) ERROR 23502 (not_null_violation) null value in column \"description_en\" of relation \"products\" violates not-null constraint\n\n    table: products\n    column: description_en\n\nFailing row contains (126ec224-ca4e-4bc5-a1a2-251d00520876, MAGP0.6.2, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, 2023-05-25 15:02:27.591724, 2023-05-25 15:02:27.591724, 14, not_synced, null).", field: nil, changeset: nil, query: nil, error_context: [], vars: [], path: [], stacktrace: #Stacktrace<>, class: :unknown}}
    (db_connection 2.5.0) lib/db_connection.ex:996: DBConnection.rollback/2
    (ash 2.9.12) lib/ash/data_layer/data_layer.ex:288: Ash.DataLayer.transaction/4
    (ash 2.9.12) lib/ash/actions/create/bulk.ex:187: anonymous fn/7 in Ash.Actions.Create.Bulk.do_run/5
    (elixir 1.14.1) lib/stream.ex:612: anonymous fn/4 in Stream.map/2
    (elixir 1.14.1) lib/enum.ex:4751: Enumerable.List.reduce/3
    (elixir 1.14.1) lib/stream.ex:1026: Stream.do_transform_inner_list/7
    (elixir 1.14.1) lib/stream.ex:1811: Enumerable.Stream.do_each/4
    (elixir 1.14.1) lib/enum.ex:4307: Enum.reduce/3
    (ash 2.9.12) lib/ash/actions/create/bulk.ex:398: Ash.Actions.Create.Bulk.do_run/5
    (ash 2.9.12) lib/ash/actions/create/bulk.ex:69: Ash.Actions.Create.Bulk.run/5
    (ash 2.9.12) lib/ash/api/api.ex:1884: Ash.Api.bulk_create!/5
    /home/tommasop/code/work/magic/mms_biztalk/lib/mms_biztalk/flows/steps/upsert_resources.ex:1: (file)

zachdaniel:

I wonder

.tommasop:

this is the collection

.tommasop:

pry(1)> input.resources_attributes
[
  %{
    category_id: "054bb31e-932d-4509-b96c-18af44abf5ad",
    data_in_id: "8b58c644-c9a4-4ff7-8314-ea8262a826fb",
    id: "73c7a25b-1bd3-45f4-bb5d-0dae1b293022",
    quantity: 14.0,
    sku_code: "MAGP0.6.2"
  }
]

zachdaniel:

I wonder if postgres is validating the insert before trying to apply the on conflict?

zachdaniel:

like “this insert would fail, and might be an insert”

zachdaniel:

can you try filling in the required fields in your input?

zachdaniel:

And seeing if that creates a row

zachdaniel:

or if it updates

.tommasop:

good idea

.tommasop:

with all the needed fields it works but it updates all the fields:

.tommasop:

p = MmsBiztalk.bulk_create!(input.resources_attributes, input.resource, :create, upsert?: true, upsert_fields: [:sku_code, :quantity], return_records?: true)
%Ash.BulkResult{
  status: :success,
  errors: [],
  records: [
    #MmsBiztalk.Product<
      category: #Ash.NotLoaded<:relationship>,
      data_in: #Ash.NotLoaded<:relationship>,
      __meta__: #Ecto.Schema.Metadata<:loaded, "products">,
      id: "1905788c-1f86-4cef-a92c-2480ede1a59f",
      sku_code: "MAGP0.6.2",
      brand: "lotus",
      description_en: "cool",
      description_full_it: nil,
      description_full_en: nil,
      description_full_es: nil,
      description_full_de: nil,
      gross_unit_weight: 1.0,
      gross_unit_weight_measurement: nil,
      price: 1.0,
      macrofamily_code: "A",
      family_code: "A",
      logistic_class: "A",
      created_at: nil,
      quantity: 14.0,
      status: :not_synced,
      inserted_at: ~U[2023-05-25 15:13:14.831601Z],
      updated_at: ~U[2023-05-25 15:13:14.831601Z],
      deleted_at: nil,
      data_in_id: nil,
      category_id: nil,
      aggregates: %{},
      calculations: %{},
      __order__: nil,
      ...
    >
  ],
  notifications: [],
  error_count: 0
}

.tommasop:

the behavior is consistent if I use id or another identity ( :unique_sku_code )

zachdaniel:

are you sure its updating all the fields and not just creating a new record?

zachdaniel:

if its updating more than the upsert_fields then something very very very strange is happening

zachdaniel:

ohhhhh

zachdaniel:

😢

zachdaniel:

I think I found it

zachdaniel:

maybe

zachdaniel:

Yeah, this isn’t your fault, sorry

zachdaniel:

actually…okay well, I found a place that is definitely problematic

zachdaniel:

and maybe we’re looking somewhere along these lines

zachdaniel:

its not a manual create action is it?

zachdaniel:

what happens if you do this in iex

Ash.DataLayer.data_layer_can?(MmsBiztalk.Product, :bulk_create)

zachdaniel:

Do you get true ?

.tommasop:

yes

zachdaniel:

😢

zachdaniel:

thought I had figured it out

zachdaniel:

I fixed something, but I don’t think it will help you

zachdaniel:

but try pulling main anyway

.tommasop:

ok thanks

zachdaniel:

If this fixes it then we have some more investigating to do as to how you even got into this case anyway. But its probably not it unfortunately

zachdaniel:

lets double check this: https://discord.com/channels/711271361523351632/1110978687203016764/1111312811247804457

zachdaniel:

i.e use a static id and do your upsert twice

zachdaniel:

can I see the :create action?

.tommasop:

    create :create do
      description ""

      argument :data_in_id, :uuid do
        allow_nil? false
      end

      argument :category_id, :uuid do
        allow_nil? false
      end

      change manage_relationship(:data_in_id, :data_in, type: :append_and_remove)
      change manage_relationship(:category_id, :category, type: :append_and_remove)
    end

zachdaniel:

Seems pretty standard

.tommasop:

it’s not upserting but creating new records

zachdaniel:

even when the id matches?

zachdaniel:

Do you have a default_accept ?

zachdaniel:

Is sku_code writable?

zachdaniel:

is id writable?

.tommasop:

sku_code is writable, and no default_accept? , id is writeable

zachdaniel:

🤔 🤔 🤔 🤔

zachdaniel:

okay, so I think I need to see like a complete script or something as well as the resource (in an ideal world, this would be reproduced in a test in ash_postgres )

.tommasop:

I can try to write a minimal reproduction test in ash_postgres

zachdaniel:

that would be amazing 🙂

.tommasop:

will do my best 🙂

frankdugan3:

I’m trying to do a bulk insert w/ a managed relationship like this:

%{
  name: "Cyberdyne Systems",
  code: "CYD",
  notes: "Develops military and civilian robotics, including Skynet.",
  type: %{
    label: "Robotics and Artificial Intelligence",
    description:
      "Focused on the development of military and civilian robotics, including the AI system Skynet."
  }
},

create :create do
  argument :type, :map, allow_nil?: false

  change manage_relationship(:type, :type,
            on_lookup: :relate,
            on_no_match: :create,
            on_match: :ignore,
            on_missing: :unrelate
          )
end

Which produces this error:

** (Ash.Error.Unknown) Unknown Error

* ** (ArgumentError) unknown field for :on_conflict, got: :type
  (ecto 3.10.1) lib/ecto/repo/schema.ex:739: Ecto.Repo.Schema.field_source!/2
  (elixir 1.14.5) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
  (elixir 1.14.5) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
  (ecto 3.10.1) lib/ecto/repo/schema.ex:703: Ecto.Repo.Schema.on_conflict/5
  (ecto 3.10.1) lib/ecto/repo/schema.ex:55: Ecto.Repo.Schema.do_insert_all/7
  (ash_postgres 1.3.28) lib/data_layer.ex:1096: AshPostgres.DataLayer.bulk_create/3
  (ash 2.9.12) lib/ash/actions/create/bulk.ex:759: Ash.Actions.Create.Bulk.run_batch/10
  (ash 2.9.12) lib/ash/actions/create/bulk.ex:229: anonymous fn/9 in Ash.Actions.Create.Bulk.do_run/5
  (ecto_sql 3.10.1) lib/ecto/adapters/sql.ex:1203: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
  (db_connection 2.5.0) lib/db_connection.ex:1630: DBConnection.run_transaction/4
  (ash 2.9.12) lib/ash/actions/create/bulk.ex:191: anonymous fn/7 in Ash.Actions.Create.Bulk.do_run/5
  (elixir 1.14.5) lib/stream.ex:612: anonymous fn/4 in Stream.map/2
  (elixir 1.14.5) lib/enum.ex:4751: Enumerable.List.reduce/3
  (elixir 1.14.5) lib/stream.ex:1026: Stream.do_transform_inner_list/7
  (elixir 1.14.5) lib/stream.ex:1813: Enumerable.Stream.do_each/4
  (elixir 1.14.5) lib/enum.ex:4307: Enum.reduce/3
  (ash 2.9.12) lib/ash/actions/create/bulk.ex:405: Ash.Actions.Create.Bulk.do_run/5
  (ash 2.9.12) lib/ash/actions/create/bulk.ex:69: Ash.Actions.Create.Bulk.run/5
  (ash 2.9.12) lib/ash/api/api.ex:1878: Ash.Api.bulk_create!/5
  priv/repo/seeds/seed.exs:131: Hsm.Seed.seed_vendors/0
    (ash 2.9.12) lib/ash/api/api.ex:1881: Ash.Api.bulk_create!/5
    priv/repo/seeds/seed.exs:131: Hsm.Seed.seed_vendors/0
    (elixir 1.14.5) lib/code.ex:1260: Code.require_file/2
    (mix 1.14.5) lib/mix/tasks/run.ex:144: Mix.Tasks.Run.run/5
    (mix 1.14.5) lib/mix/tasks/run.ex:84: Mix.Tasks.Run.run/1
    (mix 1.14.5) lib/mix/task.ex:421: anonymous fn/3 in Mix.Task.run_task/4
    (mix 1.14.5) lib/mix/task.ex:479: Mix.Task.run_alias/6
    (mix 1.14.5) lib/mix/cli.ex:84: Mix.CLI.run_task/2

Is that supported, or should I be doing separate inserts and just directly manage the :type_id ?

zachdaniel:

I think its fine, but what are you passing as opts?

zachdaniel:

it looks like something is getting type instead of type_id perhaps your identity is wrong?

frankdugan3:

        Hsm.Ash.Vendors.Vendor,
        :create,
        upsert?: true,
        upsert_identity: :code,
        upsert_fields: [
          :name,
          :notes,
          :updated_at,
          :active,
          :type
        ],
        stop_on_error?: true,
        authorize?: false,
        actor: nil,
        return_records?: true

If I use :type_id instead of :type , it doesn’t upsert the types (vendors have already been created without type) and also doesn’t create any errors.

frankdugan3:

This is the query w/ :type_id :

[debug] QUERY OK db=1.5ms
INSERT INTO "vendors" ("active","code","id","name","notes","inserted_at","updated_at") VALUES
...

zachdaniel:

Are you on the latest ash? Just released recently?

zachdaniel:

add return_errors?: true

frankdugan3:

error_count: 0, errors: [] , same otherwise.

zachdaniel:

so its doing like…nothing?

frankdugan3:

Nothing w/ the type or type_id, but all the other attributes are creating/upserting. To clarify: the types have not been created first, attempting to leverage the manage_relationship . Totally get it if that’s not supported.

zachdaniel:

It should be 🙂

zachdaniel:

Could you try to reproduce in a test on ash_postgres?

frankdugan3:

It was a simple test, you want it as a PR?

zachdaniel:

yes please ❤️

zachdaniel:

will take a look at it tomorrow

.tommasop:

I’m setting up the test and it seems the problem arises when you have attributes with allow_nil? false and without a default

.tommasop:

I will set up a test with different resources because otherwise I’ll break all other tests or let me know if there is a better way

zachdaniel:

Latest release includes multiple bulk create fixes

frankdugan3:

Updated to latest releases of everything, getting a new error now. This:

def seed_vendors() do
  %{status: :success, records: vendors} =
    Hsm.Ash.Vendors.bulk_create!(
      [
        %{
          name: "Tyrell Corporation",
          code: "TYR",
          notes: "Produces advanced androids and artificial intelligence systems.",
          type: %{
            label: "Robotics",
            description:
              "Specializes in the design and manufacturing of advanced robotic systems."
          }
        },
        # ...
      ],
      Hsm.Ash.Vendors.Vendor,
      :create,
      upsert?: true,
      upsert_identity: :name,
      upsert_fields: [
        :code,
        :notes,
        :updated_at,
        :active,
        :type_id
      ],
      stop_on_error?: true,
      authorize?: false,
      actor: nil,
      return_records?: true,
      return_errors?: true
    )
end

Results in this:

rollback []
** (Ash.Error.Unknown) Unknown Error

* 1
  (elixir 1.14.5) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
  (ash 2.9.14) lib/ash/error/error.ex:207: Ash.Error.to_error_class/2
  (ash 2.9.14) lib/ash/api/api.ex:1887: Ash.Api.bulk_create!/5
  priv/repo/seeds/seed.exs:134: Hsm.Seed.seed_vendors/0
  (elixir 1.14.5) src/elixir_compiler.erl:66: :elixir_compiler.dispatch/4
  (elixir 1.14.5) src/elixir_compiler.erl:51: :elixir_compiler.compile/3
  (elixir 1.14.5) src/elixir_compiler.erl:39: :elixir_compiler.eval_or_compile/3
  (elixir 1.14.5) src/elixir_lexical.erl:15: :elixir_lexical.run/3
  (elixir 1.14.5) src/elixir_compiler.erl:17: :elixir_compiler.quoted/3
  (elixir 1.14.5) lib/module/parallel_checker.ex:110: Module.ParallelChecker.verify/1
  (elixir 1.14.5) lib/code.ex:1260: Code.require_file/2
  (mix 1.14.5) lib/mix/tasks/run.ex:144: Mix.Tasks.Run.run/5
  (mix 1.14.5) lib/mix/tasks/run.ex:84: Mix.Tasks.Run.run/1
  (mix 1.14.5) lib/mix/task.ex:421: anonymous fn/3 in Mix.Task.run_task/4
  (mix 1.14.5) lib/mix/task.ex:479: Mix.Task.run_alias/6
  (mix 1.14.5) lib/mix/cli.ex:84: Mix.CLI.run_task/2
  (elixir 1.14.5) src/elixir_compiler.erl:66: :elixir_compiler.dispatch/4
  (elixir 1.14.5) src/elixir_compiler.erl:51: :elixir_compiler.compile/3
    (ash 2.9.14) lib/ash/api/api.ex:1887: Ash.Api.bulk_create!/5
    priv/repo/seeds/seed.exs:134: Hsm.Seed.seed_vendors/0
    (elixir 1.14.5) lib/code.ex:1260: Code.require_file/2
    (mix 1.14.5) lib/mix/tasks/run.ex:144: Mix.Tasks.Run.run/5
    (mix 1.14.5) lib/mix/tasks/run.ex:84: Mix.Tasks.Run.run/1
    (mix 1.14.5) lib/mix/task.ex:421: anonymous fn/3 in Mix.Task.run_task/4
    (mix 1.14.5) lib/mix/task.ex:479: Mix.Task.run_alias/6
    (mix 1.14.5) lib/mix/cli.ex:84: Mix.CLI.run_task/2
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed

.tommasop:

Here the error stays the same

.tommasop:

I added a test with a Manager resource which belongs to Organization the test passes for creation but fails for upsert

.tommasop:

hope the test makes sense 🙂

zachdaniel:

Will look into both tonight or tomorrow, thank you both for the tests and patience 🙂

zachdaniel:

<@1042733068168986684> alright, so, here is the deal

zachdaniel:

that is actually not supported 🙂

zachdaniel:

INSERT INTO null constraints are checked first

zachdaniel:

So because it might actually do a create, this won’t work

zachdaniel:

You’ll need to use ecto to do your update statement and/or wait until bulk updates come for Ash

zachdaniel:

However, Ash should have called that invalid before you got there actually

zachdaniel:

instead of a postgres error

zachdaniel:

I’ve fixed that issue. It was a missing required check in the bulk create logic.

zachdaniel:

<@433654314175692800> I don’t know whats going on w/ that error….seems like the error_count is somehow leaking out?

zachdaniel:

Or maybe something in here…:

  defp errors(result, invalid, opts) when is_list(invalid) do
    Enum.reduce(invalid, {result.error_count, result.errors}, fn invalid, {error_count, errors} ->
      errors(%{result | error_count: error_count, errors: errors}, invalid, opts)
    end)
  end

  defp errors(result, {:error, error}, opts) do
    if opts[:return_errors?] do
      {result.error_count + 1, [error | result.errors]}
    else
      {result.error_count + 1, []}
    end
  end

  defp errors(result, invalid, opts) do
    if Enumerable.impl_for(invalid) do
      invalid = Enum.to_list(invalid)
      errors(result, invalid, opts)
    else
      errors(result, {:error, invalid}, opts)
    end
  end

zachdaniel:

okay, found it

zachdaniel:

Something in your request is failing <@433654314175692800> but the error message was being swallowed

zachdaniel:

So it might still be broken, but we should get more info 🙂

frankdugan3:

OK, latest of of everything. This:

def seed_vendors(vendor_types) do
  %{status: :success, records: vendors, errors: errors} =
    Hsm.Ash.Vendors.bulk_create!(
      [
        %{
          name: "Tyrell Corporation",
          code: "TYR",
          type_id: Map.get(vendor_types, "Robotics"),
          notes: "Produces advanced androids and artificial intelligence systems."
        },
        # ...
      ],
  Hsm.Ash.Vendors.Vendor,
  :create,
  upsert?: true,
  upsert_identity: :name,
  upsert_fields: [
    :type_id,
    :code,
    :notes,
    :updated_at,
    :active
  ],
  stop_on_error?: true,
  authorize?: false,
  actor: nil,
  return_errors?: true,
  return_records?: true
)

Results in this:

** (FunctionClauseError) no function clause matching in Enum.flat_map_list/2

    The following arguments were given to Enum.flat_map_list/2:

        # 1
        nil

        # 2
        #Function<12.129275706/1 in Ash.Error.flatten_preserving_keywords/1>

    Attempted function clauses (showing 2 out of 2):

        defp flat_map_list([head | tail], fun)
        defp flat_map_list([], _fun)

    (elixir 1.14.5) lib/enum.ex:4248: Enum.flat_map_list/2
    (elixir 1.14.5) lib/enum.ex:4250: Enum.flat_map_list/2
    (ash 2.9.15) lib/ash/error/error.ex:205: Ash.Error.to_error_class/2
    (ash 2.9.15) lib/ash/api/api.ex:1887: Ash.Api.bulk_create!/5
    priv/repo/seeds/seed.exs:253: Hsm.Seed.seed_vendors/1
    (elixir 1.14.5) lib/code.ex:1260: Code.require_file/2
    (mix 1.14.5) lib/mix/tasks/run.ex:144: Mix.Tasks.Run.run/5
    (mix 1.14.5) lib/mix/tasks/run.ex:84: Mix.Tasks.Run.run/1
    (mix 1.14.5) lib/mix/task.ex:421: anonymous fn/3 in Mix.Task.run_task/4
    (mix 1.14.5) lib/mix/task.ex:479: Mix.Task.run_alias/6
    (mix 1.14.5) lib/mix/cli.ex:84: Mix.CLI.run_task/2

frankdugan3:

If I do this:

def seed_vendors() do
  %{status: :success, records: vendors, errors: errors} =
    Hsm.Ash.Vendors.bulk_create!(
      [
        %{
          name: "Tyrell Corporation",
          code: "TYR",
          type: %{
            label: "Robotics",
            description:
              "Specializes in the design and manufacturing of advanced robotic systems."
          },
          notes: "Produces advanced androids and artificial intelligence systems."
        },
        # ...
      ],
  Hsm.Ash.Vendors.Vendor,
  :create,
  upsert?: true,
  upsert_identity: :name,
  upsert_fields: [
    :type_id,
    :code,
    :notes,
    :updated_at,
    :active
  ],
  stop_on_error?: true,
  authorize?: false,
  actor: nil,
  return_errors?: true,
  return_records?: true
)

I actually get a proper error, and I can see I had a changeset validation problem. So I think we’re making progress on the errors! 😄

frankdugan3:

%Ash.BulkResult{
  status: :partial_success,
  errors: [
    #Ash.Changeset<
      # ...
      context: %{bulk_create: %{index: 0}},
      valid?: false
    >
    # ...
  ],
  records: [],
  notifications: [],
  error_count: 20
}

So the upsert may be working once I fix my data input. Looks like it’s just error handling that has a few edge cases.

zachdaniel:

Okay, I think I found a place that could cause errors to be something like [nil]

zachdaniel:

which would yield that error

zachdaniel:

Just pushed a change up to main that ought to help with that

frankdugan3:

OK, making progress, but ran into a new error. 😄

When doing this variety of bulk create:

def seed_vendors() do
  %{status: :success, records: vendors, errors: errors} =
    Hsm.Ash.Vendors.bulk_create!(
      [
        %{
          name: "Tyrell Corporation",
          code: "TYR",
          tax_id_type: :None,
          tax_exemption_reason: "Exempt",
          type: %{
            label: "Robotics",
            description:
              "Specializes in the design and manufacturing of advanced robotic systems."
          },
          notes: "Produces advanced androids and artificial intelligence systems."
        },

with this action:

create :create do
  argument :type, :map, allow_nil?: false

  change manage_relationship(:type, :type,
            on_lookup: :relate,
            on_no_match: :create,
            on_match: :ignore,
            on_missing: :unrelate,
            use_identities: [:id, :label]
          )
end

I get this error:

** (KeyError) key :keys not found in: nil. If you are using the dot syntax, such as map.field, make sure the left-hand side of the dot is a map
    (ash 2.9.16) lib/ash/actions/managed_relationships.ex:601: anonymous fn/2 in Ash.Actions.ManagedRelationships.pkeys/2
    (elixir 1.14.5) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
    (ash 2.9.16) lib/ash/actions/managed_relationships.ex:107: anonymous fn/4 in Ash.Actions.ManagedRelationships.setup_managed_belongs_to_relationships/3
    (elixir 1.14.5) lib/enum.ex:4751: Enumerable.List.reduce/3
    (elixir 1.14.5) lib/enum.ex:2514: Enum.reduce_while/3
    (ash 2.9.16) lib/ash/actions/managed_relationships.ex:102: Ash.Actions.ManagedRelationships.setup_managed_belongs_to_relationships/3
    (ash 2.9.16) lib/ash/actions/create/bulk.ex:728: anonymous fn/3 in Ash.Actions.Create.Bulk.run_batch/11
    (elixir 1.14.5) lib/enum.ex:4307: anonymous fn/3 in Enum.reduce/3
    (elixir 1.14.5) lib/stream.ex:1801: anonymous fn/3 in Enumerable.Stream.reduce/3
    (elixir 1.14.5) lib/enum.ex:4751: Enumerable.List.reduce/3
    (elixir 1.14.5) lib/stream.ex:1813: Enumerable.Stream.do_each/4
    (elixir 1.14.5) lib/enum.ex:4307: Enum.reduce/3
    (ash 2.9.16) lib/ash/actions/create/bulk.ex:713: Ash.Actions.Create.Bulk.run_batch/11
    (ash 2.9.16) lib/ash/actions/create/bulk.ex:225: anonymous fn/9 in Ash.Actions.Create.Bulk.do_run/5
    (ecto_sql 3.10.1) lib/ecto/adapters/sql.ex:1203: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
    (db_connection 2.5.0) lib/db_connection.ex:1630: DBConnection.run_transaction/4
    (ash 2.9.16) lib/ash/actions/create/bulk.ex:187: anonymous fn/7 in Ash.Actions.Create.Bulk.do_run/5
    (elixir 1.14.5) lib/stream.ex:612: anonymous fn/4 in Stream.map/2
    (elixir 1.14.5) lib/enum.ex:4751: Enumerable.List.reduce/3
    (elixir 1.14.5) lib/stream.ex:1026: Stream.do_transform_inner_list/7

frankdugan3:

^ This is on the latest released version.

frankdugan3:

Also, still getting this exact error for this kind of bulk create. ^

zachdaniel:

😢 I might need another test reproduction for this one

frankdugan3:

NP, will PR a few when I get a chance.

zachdaniel:

<@433654314175692800> you’re in luck, a client encountered this issue

zachdaniel:

have a fix incoming, at least for the specific issue raising on []

frankdugan3:

Oh, awesome! Sorry I hadn’t gotten around to a repro PR, was using this for optional improvements to staging seed, had some other priorities come up ahead of it. 😅

frankdugan3:

Sadly the updates to Ash/AshPostgres didn’t change anything for my two cases. Will see if I can get to a repro PR today or tomorrow. 😢

zachdaniel:

Hmmmm