Creating a field of type custom field as a BaseFieldDefinition.

As with all field types, it's possible to create a field of type custom field as a BaseFieldDefinition on an entity. The following example is for a custom field with two sub-fields of type text, First Name (first_name) and Last Name (last_name):

// The field type is 'custom'.
$fields['target_types'] = BaseFieldDefinition::create('custom')
  ->setLabel(t('Student Name'))
  ->setDescription(t('The name of the student'))
  ->setCardinality(FieldStorageDefinitionInterface::CARDINALITY_UNLIMITED)
  ->setDisplayConfigurable('view', TRUE)
  ->setDisplayConfigurable('form', TRUE)
  // Columns define the field type, in this example first_name and last_name,
  // both of which are strings.
  ->setSetting('columns', [
    'first_name' => [
      'name' => 'first_name',
      'type' => 'string',
      'length' => 128,
    ],
    'last_name' => [
      'name' => 'last_name',
      'type' => 'string',
      'length' => 128,
    ],
  ])

  // Field settings define the widget used. Each field needs to have the
  // widget defined in the field settings. In this case, first_name and lastname.
  ->setSetting('field_settings', [
    'first_name' => [
      // The value of 'type' is the ID of the widget plugin to use.
      'type' => 'text',
      'weight' => -10,
      'check_empty' => FALSE,
      'widget_settings' => [
        'label' => 'First Name',
        'translatable' => FALSE,
        'settings' => [
          'description' => 'Enter the first name of the student',
          'description_display' => 'after',
          'size' => 60,
          'placeholder' => '',
          'maxlength' => 128,
          'prefix' => '',
          'suffix' => '',
          'required' => TRUE,
        ],
      ],
    ],
    'last_name' => [
      'type' => 'text',
      'weight' => -10,
      'check_empty' => FALSE,
      'widget_settings' => [
        'label' => 'Last Name',
        'translatable' => FALSE,
        'settings' => [
          'description' => 'Enter the last name of the student',
          'description_display' => 'after',
          'size' => 60,
          'placeholder' => '',
          'maxlength' => 128,
          'prefix' => '',
          'suffix' => '',
          'required' => TRUE,
        ],
      ],
    ],
  ])
  // The form display options define how the custom field, as a whole,
  // with all elements, will be displayed in the form. Individual field
  // widget options are handled above.
  ->setDisplayOptions('form', [
    'type' => 'custom_flexbox',
    'region' => 'content',
    'settings' => [
      'label' => TRUE,
      'wrapper' => 'div',
      'open' => TRUE,
    ],
  ])
  // The view display options define how the custom field, as a whole,
  // with all elements, will be displayed on the front end.
  ->setDisplayOptions('view', [
    'type' => 'custom_formatter',
    'label' => 'above',
    'settings' => [
      'fields' => [],
    ],
    'third_party_settings' => [],
    'region' => 'content',
  ]);

The values to use in the settings can be determined as follows:

1. Add the field to your entity in the GUI, on the admin pages. Configure as necessary.
2. Export the configuration to the file system
3. Examine the exported configuration for the newly created field, and go through field.storage.[ENTITY_TYPE].[FIELD_NAME].yml, field.field.[ENTITY_TYPE]..[BUNDLE_TYPE].[FIELD_NAME].yml, core.entity_form_display.[ENTITY_TYPE].[BUNDLE_TYPE].default.yml and core.entity_view_display.[ENTITY_TYPE].[BUNDLE_TYPE].default.yml, using the relevant values from these files to populate the BaseFieldDefinition in the example above.

The main keys you'll need to focus on are columns and field_settings.