Used functions and definitions
concat_ws(sep, [str | array(str)]+) - Returns the concatenation of the strings separated by sep.
Examples:
SELECT concat_ws(’ ', ‘Spark’, ‘SQL’);
Spark SQL
collect_set(expr) - Collects and returns a set of unique elements.
Example data
We string the fourth column should be combined into one line
select
Asset_a
, concat_ws(',',collect_set(NT_Login)) as ntlogin
from (
select
Asset
,Platform
,UserOrBatch
,NT_Login
from eip_rewards_usage
where Platform='hercules'
and UserOrBatch='User'
)
group by 1
order by 1 asc
First we need to increase the limit for the data group by, and within a group of data items collect_set NT_Login together, CONCAT_WS merge, '' as a separator
sparksql Function Documentation:
https://docs.databricks.com/spark/latest/spark-sql/language-manual/functions.html#concat_ws